Python 3.9+
├── Data Processing
│   ├── pandas (1.5+)          # Data manipulation and analysis
│   ├── numpy (1.23+)          # Numerical computing
│   └── scipy (1.9+)           # Scientific computing
│
├── Machine Learning
│   ├── scikit-learn (1.2+)    # ML algorithms and preprocessing
│   ├── xgboost (1.7+)         # Gradient boosting
│   ├── lightgbm (3.3+)        # Fast gradient boosting (optional)
│   └── imbalanced-learn       # SMOTE for class imbalance
│
└── Visualization
    ├── matplotlib (3.6+)      # Plotting library
    └── seaborn (0.12+)        # Statistical visualizations

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings

# Configuration
warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette('husl')
%matplotlib inline

# Display settings for better readability
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 100)
pd.set_option('display.precision', 3)

print("Libraries loaded successfully!")

Libraries loaded successfully!

# CONFIG: Update this path to your CSV file
file_path = 'Friday-WorkingHours-Afternoon-DDos.pcap_ISCX.csv'  # <-- CHANGE THIS

def load_data(filepath):
    """
    Load CSV file with error handling and column name cleaning.
    
    Returns:
        pandas DataFrame or None if loading fails
    """
    try:
        df = pd.read_csv(filepath)
        
        # Clean column names - strip leading/trailing whitespace
        df.columns = df.columns.str.strip()
        
        print(f"✓ Data loaded successfully!")
        print(f"  Rows: {df.shape[0]:,}")
        print(f"  Columns: {df.shape[1]}")
        print(f"✓ Column names cleaned (whitespace stripped)")
        return df
    except FileNotFoundError:
        print(f"✗ Error: File '{filepath}' not found.")
        return None
    except Exception as e:
        print(f"✗ Error loading data: {e}")
        return None

# Load the data
df = load_data(file_path)

✓ Data loaded successfully!
  Rows: 225,745
  Columns: 85
✓ Column names cleaned (whitespace stripped)

def initial_inspection(df):
    """
    Perform initial inspection of the dataset.
    """
    print("=" * 80)
    print("INITIAL INSPECTION")
    print("=" * 80)
    
    print("\n📊 Dataset Shape:")
    print(f"   Rows: {df.shape[0]:,}")
    print(f"   Columns: {df.shape[1]}")
    
    print("\n📋 Column Names:")
    for i, col in enumerate(df.columns, 1):
        print(f"   {i}. {col}")
    
    print("\n🔢 Data Types:")
    print(df.dtypes)
    
    print("\n💾 Memory Usage:")
    memory_usage = df.memory_usage(deep=True).sum() / 1024**2
    print(f"   Total: {memory_usage:.2f} MB")
    
    return None

if df is not None:
    initial_inspection(df)

================================================================================
INITIAL INSPECTION
================================================================================

📊 Dataset Shape:
   Rows: 225,745
   Columns: 85

📋 Column Names:
   1. Flow ID
   2. Source IP
   3. Source Port
   4. Destination IP
   5. Destination Port
   6. Protocol
   7. Timestamp
   8. Flow Duration
   9. Total Fwd Packets
   10. Total Backward Packets
   11. Total Length of Fwd Packets
   12. Total Length of Bwd Packets
   13. Fwd Packet Length Max
   14. Fwd Packet Length Min
   15. Fwd Packet Length Mean
   16. Fwd Packet Length Std
   17. Bwd Packet Length Max
   18. Bwd Packet Length Min
   19. Bwd Packet Length Mean
   20. Bwd Packet Length Std
   21. Flow Bytes/s
   22. Flow Packets/s
   23. Flow IAT Mean
   24. Flow IAT Std
   25. Flow IAT Max
   26. Flow IAT Min
   27. Fwd IAT Total
   28. Fwd IAT Mean
   29. Fwd IAT Std
   30. Fwd IAT Max
   31. Fwd IAT Min
   32. Bwd IAT Total
   33. Bwd IAT Mean
   34. Bwd IAT Std
   35. Bwd IAT Max
   36. Bwd IAT Min
   37. Fwd PSH Flags
   38. Bwd PSH Flags
   39. Fwd URG Flags
   40. Bwd URG Flags
   41. Fwd Header Length
   42. Bwd Header Length
   43. Fwd Packets/s
   44. Bwd Packets/s
   45. Min Packet Length
   46. Max Packet Length
   47. Packet Length Mean
   48. Packet Length Std
   49. Packet Length Variance
   50. FIN Flag Count
   51. SYN Flag Count
   52. RST Flag Count
   53. PSH Flag Count
   54. ACK Flag Count
   55. URG Flag Count
   56. CWE Flag Count
   57. ECE Flag Count
   58. Down/Up Ratio
   59. Average Packet Size
   60. Avg Fwd Segment Size
   61. Avg Bwd Segment Size
   62. Fwd Header Length.1
   63. Fwd Avg Bytes/Bulk
   64. Fwd Avg Packets/Bulk
   65. Fwd Avg Bulk Rate
   66. Bwd Avg Bytes/Bulk
   67. Bwd Avg Packets/Bulk
   68. Bwd Avg Bulk Rate
   69. Subflow Fwd Packets
   70. Subflow Fwd Bytes
   71. Subflow Bwd Packets
   72. Subflow Bwd Bytes
   73. Init_Win_bytes_forward
   74. Init_Win_bytes_backward
   75. act_data_pkt_fwd
   76. min_seg_size_forward
   77. Active Mean
   78. Active Std
   79. Active Max
   80. Active Min
   81. Idle Mean
   82. Idle Std
   83. Idle Max
   84. Idle Min
   85. Label

🔢 Data Types:
Flow ID                         object
Source IP                       object
Source Port                      int64
Destination IP                  object
Destination Port                 int64
Protocol                         int64
Timestamp                       object
Flow Duration                    int64
Total Fwd Packets                int64
Total Backward Packets           int64
Total Length of Fwd Packets      int64
Total Length of Bwd Packets      int64
Fwd Packet Length Max            int64
Fwd Packet Length Min            int64
Fwd Packet Length Mean         float64
Fwd Packet Length Std          float64
Bwd Packet Length Max            int64
Bwd Packet Length Min            int64
Bwd Packet Length Mean         float64
Bwd Packet Length Std          float64
Flow Bytes/s                   float64
Flow Packets/s                 float64
Flow IAT Mean                  float64
Flow IAT Std                   float64
Flow IAT Max                     int64
Flow IAT Min                     int64
Fwd IAT Total                    int64
Fwd IAT Mean                   float64
Fwd IAT Std                    float64
Fwd IAT Max                      int64
Fwd IAT Min                      int64
Bwd IAT Total                    int64
Bwd IAT Mean                   float64
Bwd IAT Std                    float64
Bwd IAT Max                      int64
Bwd IAT Min                      int64
Fwd PSH Flags                    int64
Bwd PSH Flags                    int64
Fwd URG Flags                    int64
Bwd URG Flags                    int64
Fwd Header Length                int64
Bwd Header Length                int64
Fwd Packets/s                  float64
Bwd Packets/s                  float64
Min Packet Length                int64
Max Packet Length                int64
Packet Length Mean             float64
Packet Length Std              float64
Packet Length Variance         float64
FIN Flag Count                   int64
SYN Flag Count                   int64
RST Flag Count                   int64
PSH Flag Count                   int64
ACK Flag Count                   int64
URG Flag Count                   int64
CWE Flag Count                   int64
ECE Flag Count                   int64
Down/Up Ratio                    int64
Average Packet Size            float64
Avg Fwd Segment Size           float64
Avg Bwd Segment Size           float64
Fwd Header Length.1              int64
Fwd Avg Bytes/Bulk               int64
Fwd Avg Packets/Bulk             int64
Fwd Avg Bulk Rate                int64
Bwd Avg Bytes/Bulk               int64
Bwd Avg Packets/Bulk             int64
Bwd Avg Bulk Rate                int64
Subflow Fwd Packets              int64
Subflow Fwd Bytes                int64
Subflow Bwd Packets              int64
Subflow Bwd Bytes                int64
Init_Win_bytes_forward           int64
Init_Win_bytes_backward          int64
act_data_pkt_fwd                 int64
min_seg_size_forward             int64
Active Mean                    float64
Active Std                     float64
Active Max                       int64
Active Min                       int64
Idle Mean                      float64
Idle Std                       float64
Idle Max                         int64
Idle Min                         int64
Label                           object
dtype: object

💾 Memory Usage:
   Total: 215.79 MB

# Display first few rows
print("First 5 rows:")
display(df.head())

print("\nLast 5 rows:")
display(df.tail())

First 5 rows:

Last 5 rows:

def data_quality_check(df):
    """
    Comprehensive data quality assessment.
    """
    print("=" * 80)
    print("DATA QUALITY CHECK")
    print("=" * 80)
    
    # Missing values
    print("\n🔍 Missing Values:")
    missing = df.isnull().sum()
    missing_pct = (missing / len(df)) * 100
    missing_df = pd.DataFrame({
        'Missing_Count': missing,
        'Missing_Percentage': missing_pct
    })
    missing_df = missing_df[missing_df['Missing_Count'] > 0].sort_values('Missing_Count', ascending=False)
    
    if len(missing_df) > 0:
        print(missing_df)
    else:
        print("   ✓ No missing values detected!")
    
    # Infinite values (for numerical columns)
    print("\n♾️  Infinite Values:")
    numeric_cols = df.select_dtypes(include=[np.number]).columns
    inf_count = pd.Series(dtype=int)
    
    for col in numeric_cols:
        inf_values = np.isinf(df[col]).sum()
        if inf_values > 0:
            inf_count[col] = inf_values
    
    if len(inf_count) > 0:
        print("   ⚠️ Columns with infinite values:")
        for col, count in inf_count.items():
            pct = (count / len(df)) * 100
            print(f"   {col}: {count:,} ({pct:.2f}%)")
    else:
        print("   ✓ No infinite values detected!")
    
    # Duplicate rows
    print("\n🔄 Duplicate Rows:")
    duplicates = df.duplicated().sum()
    print(f"   Count: {duplicates:,}")
    print(f"   Percentage: {(duplicates/len(df)*100):.2f}%")
    
    # Data type summary
    print("\n📊 Data Type Summary:")
    dtype_counts = df.dtypes.value_counts()
    for dtype, count in dtype_counts.items():
        print(f"   {dtype}: {count} columns")
    
    return missing_df

if df is not None:
    missing_summary = data_quality_check(df)

================================================================================
DATA QUALITY CHECK
================================================================================

🔍 Missing Values:
              Missing_Count  Missing_Percentage
Flow Bytes/s              4               0.002

♾️  Infinite Values:
   ⚠️ Columns with infinite values:
   Flow Bytes/s: 30 (0.01%)
   Flow Packets/s: 34 (0.02%)

🔄 Duplicate Rows:

   Count: 2
   Percentage: 0.00%

📊 Data Type Summary:
   int64: 56 columns
   float64: 24 columns
   object: 5 columns

def statistical_summary(df):
    """
    Generate statistical summaries for numerical and categorical features.
    """
    print("=" * 80)
    print("STATISTICAL SUMMARY")
    print("=" * 80)
    
    # Identify column types
    numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
    categorical_cols = df.select_dtypes(include=['object', 'category']).columns.tolist()
    
    print(f"\n📈 Numerical Features: {len(numeric_cols)}")
    print(f"📝 Categorical Features: {len(categorical_cols)}")
    
    # Numerical summary
    if numeric_cols:
        print("\n" + "="*80)
        print("NUMERICAL FEATURES SUMMARY")
        print("="*80)
        display(df[numeric_cols].describe())
    
    # Categorical summary
    if categorical_cols:
        print("\n" + "="*80)
        print("CATEGORICAL FEATURES SUMMARY")
        print("="*80)
        for col in categorical_cols:
            print(f"\n{col}:")
            print(f"  Unique values: {df[col].nunique()}")
            print(f"  Most common:")
            print(df[col].value_counts().head(5))
            print("-" * 40)
    
    return numeric_cols, categorical_cols

if df is not None:
    numeric_cols, categorical_cols = statistical_summary(df)

================================================================================
STATISTICAL SUMMARY
================================================================================

📈 Numerical Features: 80
📝 Categorical Features: 5

================================================================================
NUMERICAL FEATURES SUMMARY
================================================================================

================================================================================
CATEGORICAL FEATURES SUMMARY
================================================================================

Flow ID:
  Unique values: 86421
  Most common:
Flow ID
8.0.6.4-8.6.0.1-0-0-0                      44
192.168.10.25-17.253.14.125-123-123-17     44
192.168.10.255-192.168.10.3-137-137-17     24
192.168.10.19-192.168.10.50-137-137-17     19
192.168.10.16-192.168.10.50-48318-139-6    19
Name: count, dtype: int64
----------------------------------------

Source IP:
  Unique values: 2067
  Most common:
Source IP
172.16.0.1       128181
192.168.10.50     32896
192.168.10.15      9278
192.168.10.12      9216
192.168.10.3       8692
Name: count, dtype: int64
----------------------------------------

Destination IP:
  Unique values: 2554
  Most common:
Destination IP
192.168.10.50    128834
172.16.0.1        31343
192.168.10.3      24165
192.168.10.1       8521
192.168.10.15      2048
Name: count, dtype: int64
----------------------------------------

Timestamp:
  Unique values: 93
  Most common:
Timestamp
7/7/2017 4:13    11188
7/7/2017 3:57    10769
7/7/2017 3:58     9731
7/7/2017 3:59     9460
7/7/2017 4:11     9438
Name: count, dtype: int64
----------------------------------------

Label:
  Unique values: 2
  Most common:
Label
DDoS      128027
BENIGN     97718
Name: count, dtype: int64
----------------------------------------

def visualize_missing_data(df):
    """
    Create visualizations for missing data patterns.
    """
    missing = df.isnull().sum()
    missing = missing[missing > 0].sort_values(ascending=False)
    
    if len(missing) == 0:
        print("✓ No missing data to visualize!")
        return
    
    fig, axes = plt.subplots(1, 2, figsize=(15, 5))
    
    # Bar plot of missing values
    axes[0].barh(range(len(missing)), missing.values)
    axes[0].set_yticks(range(len(missing)))
    axes[0].set_yticklabels(missing.index)
    axes[0].set_xlabel('Number of Missing Values')
    axes[0].set_title('Missing Values by Feature')
    axes[0].invert_yaxis()
    
    # Heatmap of missing data patterns
    missing_matrix = df[missing.index].isnull().astype(int)
    sns.heatmap(missing_matrix.T, cbar=True, cmap='YlOrRd', 
                yticklabels=missing.index, ax=axes[1])
    axes[1].set_title('Missing Data Pattern\n(Yellow = Missing)')
    axes[1].set_xlabel('Row Index')
    
    plt.tight_layout()
    plt.show()

if df is not None:
    visualize_missing_data(df)

def plot_distributions(df, numeric_cols):
    """
    Create distribution plots for numerical features.
    Handles NaN and infinite values automatically.
    """
    if not numeric_cols:
        print("No numerical columns to plot.")
        return
    
    n_cols = min(len(numeric_cols), 4)
    n_rows = (len(numeric_cols) - 1) // n_cols + 1
    
    fig, axes = plt.subplots(n_rows, n_cols, figsize=(20, 5*n_rows))
    axes = axes.flatten() if n_rows * n_cols > 1 else [axes]
    
    for idx, col in enumerate(numeric_cols):
        # Remove NaN and infinite values for plotting
        data = df[col].replace([np.inf, -np.inf], np.nan).dropna()
        
        # Check if we have any valid data to plot
        if len(data) == 0:
            axes[idx].text(0.5, 0.5, f'{col}\n(No finite values)', 
                          ha='center', va='center', fontsize=12, 
                          transform=axes[idx].transAxes)
            axes[idx].set_xticks([])
            axes[idx].set_yticks([])
            continue
        
        # Histogram
        axes[idx].hist(data, bins=30, alpha=0.7, edgecolor='black')
        
        # Title with sample size info
        inf_count = np.isinf(df[col]).sum()
        title = f'{col}\n(n={len(data):,}'
        if inf_count > 0:
            title += f', {inf_count} inf values excluded)'
        else:
            title += ')'
        axes[idx].set_title(title, fontsize=12, fontweight='bold')
        
        axes[idx].set_xlabel('Value')
        axes[idx].set_ylabel('Frequency')
        
        # Add mean and median lines
        mean_val = data.mean()
        median_val = data.median()
        axes[idx].axvline(mean_val, color='red', linestyle='--', linewidth=2, 
                         label=f'Mean: {mean_val:.2f}')
        axes[idx].axvline(median_val, color='green', linestyle='--', linewidth=2, 
                         label=f'Median: {median_val:.2f}')
        axes[idx].legend()
    
    # Hide empty subplots
    for idx in range(len(numeric_cols), len(axes)):
        axes[idx].axis('off')
    
    plt.tight_layout()
    plt.show()

if df is not None and numeric_cols:
    print("=" * 80)
    print("DISTRIBUTION ANALYSIS")
    print("=" * 80)
    plot_distributions(df, numeric_cols)

================================================================================
DISTRIBUTION ANALYSIS
================================================================================

def plot_boxplots(df, numeric_cols):
    """
    Create box plots for outlier detection.
    Handles NaN and infinite values automatically.
    """
    if not numeric_cols:
        print("No numerical columns to plot.")
        return
    
    n_cols = min(len(numeric_cols), 4)
    n_rows = (len(numeric_cols) - 1) // n_cols + 1
    
    fig, axes = plt.subplots(n_rows, n_cols, figsize=(20, 5*n_rows))
    axes = axes.flatten() if n_rows * n_cols > 1 else [axes]
    
    for idx, col in enumerate(numeric_cols):
        # Remove NaN and infinite values
        data = df[col].replace([np.inf, -np.inf], np.nan).dropna()
        
        if len(data) == 0:
            axes[idx].text(0.5, 0.5, f'{col}\n(No finite values)', 
                          ha='center', va='center', fontsize=12,
                          transform=axes[idx].transAxes)
            axes[idx].set_xticks([])
            axes[idx].set_yticks([])
            continue
        
        axes[idx].boxplot(data, vert=True)
        axes[idx].set_title(col, fontsize=12, fontweight='bold')
        axes[idx].set_ylabel('Value')
        axes[idx].grid(True, alpha=0.3)
    
    # Hide empty subplots
    for idx in range(len(numeric_cols), len(axes)):
        axes[idx].axis('off')
    
    plt.tight_layout()
    plt.show()

if df is not None and numeric_cols:
    print("\nBOX PLOTS - Outlier Detection")
    print("=" * 80)
    plot_boxplots(df, numeric_cols)

BOX PLOTS - Outlier Detection
================================================================================

def correlation_analysis(df, numeric_cols):
    """
    Analyze and visualize correlations between numerical features.
    Handles infinite values by excluding them from correlation calculation.
    """
    if len(numeric_cols) < 2:
        print("Need at least 2 numerical columns for correlation analysis.")
        return None
    
    print("=" * 80)
    print("CORRELATION ANALYSIS")
    print("=" * 80)
    
    # Replace infinite values with NaN for correlation calculation
    df_clean = df[numeric_cols].replace([np.inf, -np.inf], np.nan)
    
    # Calculate correlation matrix
    corr_matrix = df_clean.corr()
    
    # Create correlation heatmap with improved readability
    n_features = len(numeric_cols)
    
    # Adjust figure size based on number of features
    figsize = max(12, n_features * 0.8)
    
    # Decide whether to show annotations based on number of features
    show_annot = n_features <= 15  # Only show values if 15 or fewer features
    annot_size = max(6, 12 - n_features // 3)  # Smaller text for more features
    
    plt.figure(figsize=(figsize, figsize * 0.9))
    sns.heatmap(corr_matrix, 
                annot=show_annot, 
                fmt='.2f', 
                cmap='coolwarm', 
                center=0, 
                square=True, 
                linewidths=0.5,
                cbar_kws={"shrink": 0.8},
                annot_kws={'size': annot_size})
    plt.title('Correlation Matrix Heatmap\n(Infinite values excluded)', 
              fontsize=16, fontweight='bold', pad=20)
    plt.xticks(rotation=45, ha='right', fontsize=10)
    plt.yticks(rotation=0, fontsize=10)
    plt.tight_layout()
    plt.show()
    
    if not show_annot:
        print("\n⚠️ Note: Correlation values hidden in heatmap due to many features.")
        print("   See the detailed pairs list below for specific values.\n")
    
    # Find highly correlated pairs
    print("\n🔗 Highly Correlated Feature Pairs (|correlation| > 0.7):")
    high_corr = []
    for i in range(len(corr_matrix.columns)):
        for j in range(i+1, len(corr_matrix.columns)):
            if abs(corr_matrix.iloc[i, j]) > 0.7:
                high_corr.append({
                    'Feature 1': corr_matrix.columns[i],
                    'Feature 2': corr_matrix.columns[j],
                    'Correlation': corr_matrix.iloc[i, j]
                })
    
    if high_corr:
        high_corr_df = pd.DataFrame(high_corr).sort_values('Correlation', key=abs, ascending=False)
        
        # Display ALL rows without truncation
        print(f"   Found {len(high_corr_df)} highly correlated pairs:\n")
        
        # Temporarily change display settings to show all rows
        with pd.option_context('display.max_rows', None, 'display.max_columns', None):
            display(high_corr_df)
    else:
        print("   No highly correlated pairs found.")
    
    return corr_matrix

if df is not None and len(numeric_cols) >= 2:
    corr_matrix = correlation_analysis(df, numeric_cols)

================================================================================
CORRELATION ANALYSIS
================================================================================

⚠️ Note: Correlation values hidden in heatmap due to many features.
   See the detailed pairs list below for specific values.


🔗 Highly Correlated Feature Pairs (|correlation| > 0.7):
   Found 167 highly correlated pairs:

# # Customize this section as needed
# if df is not None and len(numeric_cols) >= 2:
    
#     # Option 1: View ALL correlations in a table
#     print("ALL FEATURE CORRELATIONS (sorted by absolute value)")
#     print("=" * 80)
    
#     # Get correlation matrix
#     df_clean = df[numeric_cols].replace([np.inf, -np.inf], np.nan)
#     corr_matrix = df_clean.corr()
    
#     # Extract all unique pairs
#     all_corrs = []
#     for i in range(len(corr_matrix.columns)):
#         for j in range(i+1, len(corr_matrix.columns)):
#             all_corrs.append({
#                 'Feature 1': corr_matrix.columns[i],
#                 'Feature 2': corr_matrix.columns[j],
#                 'Correlation': corr_matrix.iloc[i, j]
#             })
    
#     all_corrs_df = pd.DataFrame(all_corrs).sort_values('Correlation', key=abs, ascending=False)
    
#     # Display without truncation
#     with pd.option_context('display.max_rows', None):
#         display(all_corrs_df)
    
#     print(f"\nTotal feature pairs: {len(all_corrs_df)}")
    
#     # Option 2: Filter by custom threshold (uncomment and modify as needed)
#     # custom_threshold = 0.5  # Change this value
#     # filtered_corrs = all_corrs_df[abs(all_corrs_df['Correlation']) > custom_threshold]
#     # print(f"\nPairs with |correlation| > {custom_threshold}:")
#     # with pd.option_context('display.max_rows', None):
#     #     display(filtered_corrs)

def generate_report(df, numeric_cols, categorical_cols):
    """
    Generate a comprehensive EDA report.
    """
    print("=" * 80)
    print("EXPLORATORY DATA ANALYSIS REPORT")
    print("=" * 80)
    
    print("\n📋 DATASET OVERVIEW")
    print("-" * 80)
    print(f"Total Rows: {df.shape[0]:,}")
    print(f"Total Columns: {df.shape[1]}")
    print(f"Numerical Features: {len(numeric_cols)}")
    print(f"Categorical Features: {len(categorical_cols)}")
    
    print("\n🔍 DATA QUALITY")
    print("-" * 80)
    total_missing = df.isnull().sum().sum()
    missing_pct = (total_missing / (df.shape[0] * df.shape[1])) * 100
    print(f"Total Missing Values: {total_missing:,} ({missing_pct:.2f}%)")
    print(f"Duplicate Rows: {df.duplicated().sum():,}")
    
    # Check for infinite values
    total_inf = 0
    for col in numeric_cols:
        total_inf += np.isinf(df[col]).sum()
    print(f"Total Infinite Values: {total_inf:,}")
    
    if total_missing > 0:
        print("\nColumns with Missing Data:")
        missing_cols = df.isnull().sum()[df.isnull().sum() > 0].sort_values(ascending=False)
        for col, count in missing_cols.items():
            pct = (count / len(df)) * 100
            print(f"  • {col}: {count:,} ({pct:.1f}%)")
    
    if total_inf > 0:
        print("\nColumns with Infinite Values:")
        for col in numeric_cols:
            inf_count = np.isinf(df[col]).sum()
            if inf_count > 0:
                pct = (inf_count / len(df)) * 100
                print(f"  • {col}: {inf_count:,} ({pct:.1f}%)")
    
    print("\n📊 NUMERICAL FEATURES SUMMARY")
    print("-" * 80)
    if numeric_cols:
        for col in numeric_cols:
            # Clean data for statistics
            data = df[col].replace([np.inf, -np.inf], np.nan).dropna()
            
            if len(data) == 0:
                print(f"\n{col}: No finite values available")
                continue
                
            print(f"\n{col}:")
            print(f"  Mean: {data.mean():.3f}")
            print(f"  Median: {data.median():.3f}")
            print(f"  Std Dev: {data.std():.3f}")
            print(f"  Range: [{data.min():.3f}, {data.max():.3f}]")
            
            # Check for outliers using IQR method
            Q1 = data.quantile(0.25)
            Q3 = data.quantile(0.75)
            IQR = Q3 - Q1
            outliers = ((data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))).sum()
            print(f"  Potential Outliers: {outliers} ({(outliers/len(data)*100):.1f}%)")
    else:
        print("No numerical features found.")
    
    print("\n📝 CATEGORICAL FEATURES SUMMARY")
    print("-" * 80)
    if categorical_cols:
        for col in categorical_cols:
            print(f"\n{col}:")
            print(f"  Unique Values: {df[col].nunique()}")
            print(f"  Most Common: {df[col].mode().values[0] if len(df[col].mode()) > 0 else 'N/A'}")
    else:
        print("No categorical features found.")
    
    print("\n" + "=" * 80)
    print("KEY RECOMMENDATIONS")
    print("=" * 80)
    
    recommendations = []
    
    if total_missing > 0:
        recommendations.append("• Handle missing values through imputation or removal")
    
    if total_inf > 0:
        recommendations.append("• Address infinite values (replace with NaN or cap at max/min)")
    
    if df.duplicated().sum() > 0:
        recommendations.append("• Consider removing or investigating duplicate rows")
    
    if numeric_cols:
        for col in numeric_cols:
            data = df[col].replace([np.inf, -np.inf], np.nan).dropna()
            if len(data) > 0:
                if data.std() / data.mean() > 2 if data.mean() != 0 else False:
                    recommendations.append(f"• Consider scaling/normalizing '{col}' (high variance)")
    
    if not recommendations:
        recommendations.append("• Dataset appears to be in good shape for analysis!")
    
    for rec in recommendations:
        print(rec)
    
    print("\n" + "=" * 80)

if df is not None:
    generate_report(df, numeric_cols, categorical_cols)

================================================================================
EXPLORATORY DATA ANALYSIS REPORT
================================================================================

📋 DATASET OVERVIEW
--------------------------------------------------------------------------------
Total Rows: 225,745
Total Columns: 85
Numerical Features: 80
Categorical Features: 5

🔍 DATA QUALITY
--------------------------------------------------------------------------------
Total Missing Values: 4 (0.00%)
Duplicate Rows: 2
Total Infinite Values: 64

Columns with Missing Data:
  • Flow Bytes/s: 4 (0.0%)

Columns with Infinite Values:
  • Flow Bytes/s: 30 (0.0%)
  • Flow Packets/s: 34 (0.0%)

📊 NUMERICAL FEATURES SUMMARY
--------------------------------------------------------------------------------

Source Port:
  Mean: 38257.568
  Median: 49799.000
  Std Dev: 23057.302
  Range: [0.000, 65534.000]
  Potential Outliers: 0 (0.0%)

Destination Port:
  Mean: 8879.619
  Median: 80.000
  Std Dev: 19754.647
  Range: [0.000, 65532.000]
  Potential Outliers: 88794 (39.3%)

Protocol:
  Mean: 7.600
  Median: 6.000
  Std Dev: 3.882
  Range: [0.000, 17.000]
  Potential Outliers: 32925 (14.6%)

Flow Duration:
  Mean: 16241648.528
  Median: 1452333.000
  Std Dev: 31524374.232
  Range: [-1.000, 119999937.000]
  Potential Outliers: 37465 (16.6%)

Total Fwd Packets:
  Mean: 4.875
  Median: 3.000
  Std Dev: 15.423
  Range: [1.000, 1932.000]
  Potential Outliers: 8705 (3.9%)

Total Backward Packets:
  Mean: 4.573
  Median: 4.000
  Std Dev: 21.755
  Range: [0.000, 2942.000]
  Potential Outliers: 7026 (3.1%)

Total Length of Fwd Packets:
  Mean: 939.463
  Median: 30.000
  Std Dev: 3249.403
  Range: [0.000, 183012.000]
  Potential Outliers: 37478 (16.6%)

Total Length of Bwd Packets:
  Mean: 5960.477
  Median: 164.000
  Std Dev: 39218.337
  Range: [0.000, 5172346.000]
  Potential Outliers: 1800 (0.8%)

Fwd Packet Length Max:
  Mean: 538.536
  Median: 20.000
  Std Dev: 1864.129
  Range: [0.000, 11680.000]
  Potential Outliers: 33009 (14.6%)

Fwd Packet Length Min:
  Mean: 27.882
  Median: 0.000
  Std Dev: 163.324
  Range: [0.000, 1472.000]
  Potential Outliers: 36226 (16.0%)

Fwd Packet Length Mean:
  Mean: 164.827
  Median: 8.667
  Std Dev: 504.893
  Range: [0.000, 3867.000]
  Potential Outliers: 26537 (11.8%)

Fwd Packet Length Std:
  Mean: 214.907
  Median: 5.302
  Std Dev: 797.411
  Range: [0.000, 6692.645]
  Potential Outliers: 29382 (13.0%)

Bwd Packet Length Max:
  Mean: 2735.585
  Median: 99.000
  Std Dev: 3705.123
  Range: [0.000, 11680.000]
  Potential Outliers: 0 (0.0%)

Bwd Packet Length Min:
  Mean: 16.719
  Median: 0.000
  Std Dev: 50.481
  Range: [0.000, 1460.000]
  Potential Outliers: 33758 (15.0%)

Bwd Packet Length Mean:
  Mean: 890.537
  Median: 92.000
  Std Dev: 1120.325
  Range: [0.000, 5800.500]
  Potential Outliers: 1 (0.0%)

Bwd Packet Length Std:
  Mean: 1230.173
  Median: 2.449
  Std Dev: 1733.201
  Range: [0.000, 8194.660]
  Potential Outliers: 299 (0.1%)

Flow Bytes/s:
  Mean: 585393.877
  Median: 1133.466
  Std Dev: 16885519.734
  Range: [-12000000.000, 2070000000.000]
  Potential Outliers: 46394 (20.6%)

Flow Packets/s:
  Mean: 14241.201
  Median: 5.175
  Std Dev: 115104.240
  Range: [-2000000.000, 3000000.000]
  Potential Outliers: 39133 (17.3%)

Flow IAT Mean:
  Mean: 1580587.237
  Median: 224516.857
  Std Dev: 2701595.785
  Range: [-1.000, 107000000.000]
  Potential Outliers: 30789 (13.6%)

Flow IAT Std:
  Mean: 4248569.381
  Median: 564167.630
  Std Dev: 7622819.077
  Range: [0.000, 69200000.000]
  Potential Outliers: 32468 (14.4%)

Flow IAT Max:
  Mean: 13489773.824
  Median: 1422624.000
  Std Dev: 26701716.686
  Range: [-1.000, 120000000.000]
  Potential Outliers: 33333 (14.8%)

Flow IAT Min:
  Mean: 28118.548
  Median: 3.000
  Std Dev: 759810.041
  Range: [-12.000, 107000000.000]
  Potential Outliers: 32148 (14.2%)

Fwd IAT Total:
  Mean: 15396522.996
  Median: 23710.000
  Std Dev: 31608257.665
  Range: [0.000, 120000000.000]
  Potential Outliers: 37265 (16.5%)

Fwd IAT Mean:
  Mean: 2540609.509
  Median: 10329.333
  Std Dev: 5934694.002
  Range: [0.000, 120000000.000]
  Potential Outliers: 34552 (15.3%)

Fwd IAT Std:
  Mean: 5195207.381
  Median: 12734.993
  Std Dev: 10786352.443
  Range: [0.000, 76700000.000]
  Potential Outliers: 31945 (14.2%)

Fwd IAT Max:
  Mean: 12994339.104
  Median: 23028.000
  Std Dev: 27488695.815
  Range: [0.000, 120000000.000]
  Potential Outliers: 32890 (14.6%)

Fwd IAT Min:
  Mean: 207369.827
  Median: 4.000
  Std Dev: 3795227.554
  Range: [-12.000, 120000000.000]
  Potential Outliers: 28078 (12.4%)

Bwd IAT Total:
  Mean: 6564701.087
  Median: 41101.000
  Std Dev: 21984549.331
  Range: [0.000, 120000000.000]
  Potential Outliers: 31473 (13.9%)

Bwd IAT Mean:
  Mean: 947632.220
  Median: 10009.400
  Std Dev: 4586373.976
  Range: [0.000, 120000000.000]
  Potential Outliers: 30429 (13.5%)

Bwd IAT Std:
  Mean: 1610306.243
  Median: 14703.719
  Std Dev: 5475777.673
  Range: [0.000, 76700000.000]
  Potential Outliers: 29913 (13.3%)

Bwd IAT Max:
  Mean: 4567514.462
  Median: 33365.000
  Std Dev: 16178651.194
  Range: [0.000, 120000000.000]
  Potential Outliers: 31246 (13.8%)

Bwd IAT Min:
  Mean: 225781.744
  Median: 3.000
  Std Dev: 4019289.759
  Range: [0.000, 120000000.000]
  Potential Outliers: 35504 (15.7%)

Fwd PSH Flags:
  Mean: 0.033
  Median: 0.000
  Std Dev: 0.179
  Range: [0.000, 1.000]
  Potential Outliers: 7500 (3.3%)

Bwd PSH Flags:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Fwd URG Flags:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Bwd URG Flags:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Fwd Header Length:
  Mean: 111.523
  Median: 72.000
  Std Dev: 375.791
  Range: [0.000, 39396.000]
  Potential Outliers: 35999 (15.9%)

Bwd Header Length:
  Mean: 106.789
  Median: 92.000
  Std Dev: 511.766
  Range: [0.000, 58852.000]
  Potential Outliers: 7329 (3.2%)

Fwd Packets/s:
  Mean: 12615.083
  Median: 2.316
  Std Dev: 110670.136
  Range: [0.000, 3000000.000]
  Potential Outliers: 39806 (17.6%)

Bwd Packets/s:
  Mean: 1641.693
  Median: 1.483
  Std Dev: 19895.934
  Range: [0.000, 2000000.000]
  Potential Outliers: 40923 (18.1%)

Min Packet Length:
  Mean: 8.073
  Median: 0.000
  Std Dev: 15.768
  Range: [0.000, 337.000]
  Potential Outliers: 32927 (14.6%)

Max Packet Length:
  Mean: 3226.045
  Median: 513.000
  Std Dev: 3813.135
  Range: [0.000, 11680.000]
  Potential Outliers: 0 (0.0%)

Packet Length Mean:
  Mean: 515.002
  Median: 110.333
  Std Dev: 559.064
  Range: [0.000, 1936.833]
  Potential Outliers: 0 (0.0%)

Packet Length Std:
  Mean: 1085.593
  Median: 154.031
  Std Dev: 1269.559
  Range: [0.000, 4731.522]
  Potential Outliers: 0 (0.0%)

Packet Length Variance:
  Mean: 2789905.545
  Median: 23725.491
  Std Dev: 4115940.963
  Range: [0.000, 22400000.000]
  Potential Outliers: 8977 (4.0%)

FIN Flag Count:
  Mean: 0.003
  Median: 0.000
  Std Dev: 0.052
  Range: [0.000, 1.000]
  Potential Outliers: 603 (0.3%)

SYN Flag Count:
  Mean: 0.033
  Median: 0.000
  Std Dev: 0.179
  Range: [0.000, 1.000]
  Potential Outliers: 7500 (3.3%)

RST Flag Count:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.011
  Range: [0.000, 1.000]
  Potential Outliers: 27 (0.0%)

PSH Flag Count:
  Mean: 0.351
  Median: 0.000
  Std Dev: 0.477
  Range: [0.000, 1.000]
  Potential Outliers: 0 (0.0%)

ACK Flag Count:
  Mean: 0.504
  Median: 1.000
  Std Dev: 0.500
  Range: [0.000, 1.000]
  Potential Outliers: 0 (0.0%)

URG Flag Count:
  Mean: 0.141
  Median: 0.000
  Std Dev: 0.348
  Range: [0.000, 1.000]
  Potential Outliers: 31774 (14.1%)

CWE Flag Count:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

ECE Flag Count:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.011
  Range: [0.000, 1.000]
  Potential Outliers: 27 (0.0%)

Down/Up Ratio:
  Mean: 1.006
  Median: 1.000
  Std Dev: 1.431
  Range: [0.000, 7.000]
  Potential Outliers: 17778 (7.9%)

Average Packet Size:
  Mean: 574.569
  Median: 141.000
  Std Dev: 626.096
  Range: [0.000, 2528.000]
  Potential Outliers: 0 (0.0%)

Avg Fwd Segment Size:
  Mean: 164.827
  Median: 8.667
  Std Dev: 504.893
  Range: [0.000, 3867.000]
  Potential Outliers: 26537 (11.8%)

Avg Bwd Segment Size:
  Mean: 890.537
  Median: 92.000
  Std Dev: 1120.325
  Range: [0.000, 5800.500]
  Potential Outliers: 1 (0.0%)

Fwd Header Length.1:
  Mean: 111.523
  Median: 72.000
  Std Dev: 375.791
  Range: [0.000, 39396.000]
  Potential Outliers: 35999 (15.9%)

Fwd Avg Bytes/Bulk:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Fwd Avg Packets/Bulk:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Fwd Avg Bulk Rate:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Bwd Avg Bytes/Bulk:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Bwd Avg Packets/Bulk:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Bwd Avg Bulk Rate:
  Mean: 0.000
  Median: 0.000
  Std Dev: 0.000
  Range: [0.000, 0.000]
  Potential Outliers: 0 (0.0%)

Subflow Fwd Packets:
  Mean: 4.875
  Median: 3.000
  Std Dev: 15.423
  Range: [1.000, 1932.000]
  Potential Outliers: 8705 (3.9%)

Subflow Fwd Bytes:
  Mean: 939.463
  Median: 30.000
  Std Dev: 3249.403
  Range: [0.000, 183012.000]
  Potential Outliers: 37478 (16.6%)

Subflow Bwd Packets:
  Mean: 4.573
  Median: 4.000
  Std Dev: 21.755
  Range: [0.000, 2942.000]
  Potential Outliers: 7026 (3.1%)

Subflow Bwd Bytes:
  Mean: 5960.477
  Median: 164.000
  Std Dev: 39218.337
  Range: [0.000, 5172346.000]
  Potential Outliers: 1800 (0.8%)

Init_Win_bytes_forward:
  Mean: 4247.437
  Median: 256.000
  Std Dev: 8037.781
  Range: [-1.000, 65535.000]
  Potential Outliers: 8926 (4.0%)

Init_Win_bytes_backward:
  Mean: 601.049
  Median: 126.000
  Std Dev: 4319.720
  Range: [-1.000, 65535.000]
  Potential Outliers: 7360 (3.3%)

act_data_pkt_fwd:
  Mean: 3.311
  Median: 2.000
  Std Dev: 12.270
  Range: [0.000, 1931.000]
  Potential Outliers: 6688 (3.0%)

min_seg_size_forward:
  Mean: 21.483
  Median: 20.000
  Std Dev: 4.167
  Range: [0.000, 52.000]
  Potential Outliers: 26655 (11.8%)

Active Mean:
  Mean: 184826.093
  Median: 0.000
  Std Dev: 797925.042
  Range: [0.000, 100000000.000]
  Potential Outliers: 43738 (19.4%)

Active Std:
  Mean: 12934.358
  Median: 0.000
  Std Dev: 210273.670
  Range: [0.000, 39500000.000]
  Potential Outliers: 5686 (2.5%)

Active Max:
  Mean: 208084.869
  Median: 0.000
  Std Dev: 900234.987
  Range: [0.000, 100000000.000]
  Potential Outliers: 43738 (19.4%)

Active Min:
  Mean: 177620.089
  Median: 0.000
  Std Dev: 784260.220
  Range: [0.000, 100000000.000]
  Potential Outliers: 43648 (19.3%)

Idle Mean:
  Mean: 10322143.419
  Median: 0.000
  Std Dev: 21853028.149
  Range: [0.000, 120000000.000]
  Potential Outliers: 33057 (14.6%)

Idle Std:
  Mean: 3611942.927
  Median: 0.000
  Std Dev: 12756893.458
  Range: [0.000, 65300000.000]
  Potential Outliers: 22539 (10.0%)

Idle Max:
  Mean: 12878128.944
  Median: 0.000
  Std Dev: 26921263.646
  Range: [0.000, 120000000.000]
  Potential Outliers: 33100 (14.7%)

Idle Min:
  Mean: 7755355.093
  Median: 0.000
  Std Dev: 19831094.450
  Range: [0.000, 120000000.000]
  Potential Outliers: 16052 (7.1%)

📝 CATEGORICAL FEATURES SUMMARY
--------------------------------------------------------------------------------

Flow ID:
  Unique Values: 86421
  Most Common: 192.168.10.25-17.253.14.125-123-123-17

Source IP:
  Unique Values: 2067
  Most Common: 172.16.0.1

Destination IP:
  Unique Values: 2554
  Most Common: 192.168.10.50

Timestamp:
  Unique Values: 93
  Most Common: 7/7/2017 4:13

Label:
  Unique Values: 2
  Most Common: DDoS

================================================================================
KEY RECOMMENDATIONS
================================================================================
• Handle missing values through imputation or removal
• Address infinite values (replace with NaN or cap at max/min)
• Consider removing or investigating duplicate rows
• Consider scaling/normalizing 'Destination Port' (high variance)
• Consider scaling/normalizing 'Total Fwd Packets' (high variance)
• Consider scaling/normalizing 'Total Backward Packets' (high variance)
• Consider scaling/normalizing 'Total Length of Fwd Packets' (high variance)
• Consider scaling/normalizing 'Total Length of Bwd Packets' (high variance)
• Consider scaling/normalizing 'Fwd Packet Length Max' (high variance)
• Consider scaling/normalizing 'Fwd Packet Length Min' (high variance)
• Consider scaling/normalizing 'Fwd Packet Length Mean' (high variance)
• Consider scaling/normalizing 'Fwd Packet Length Std' (high variance)
• Consider scaling/normalizing 'Bwd Packet Length Min' (high variance)
• Consider scaling/normalizing 'Flow Bytes/s' (high variance)
• Consider scaling/normalizing 'Flow Packets/s' (high variance)
• Consider scaling/normalizing 'Flow IAT Min' (high variance)
• Consider scaling/normalizing 'Fwd IAT Total' (high variance)
• Consider scaling/normalizing 'Fwd IAT Mean' (high variance)
• Consider scaling/normalizing 'Fwd IAT Std' (high variance)
• Consider scaling/normalizing 'Fwd IAT Max' (high variance)
• Consider scaling/normalizing 'Fwd IAT Min' (high variance)
• Consider scaling/normalizing 'Bwd IAT Total' (high variance)
• Consider scaling/normalizing 'Bwd IAT Mean' (high variance)
• Consider scaling/normalizing 'Bwd IAT Std' (high variance)
• Consider scaling/normalizing 'Bwd IAT Max' (high variance)
• Consider scaling/normalizing 'Bwd IAT Min' (high variance)
• Consider scaling/normalizing 'Fwd PSH Flags' (high variance)
• Consider scaling/normalizing 'Fwd Header Length' (high variance)
• Consider scaling/normalizing 'Bwd Header Length' (high variance)
• Consider scaling/normalizing 'Fwd Packets/s' (high variance)
• Consider scaling/normalizing 'Bwd Packets/s' (high variance)
• Consider scaling/normalizing 'FIN Flag Count' (high variance)
• Consider scaling/normalizing 'SYN Flag Count' (high variance)
• Consider scaling/normalizing 'RST Flag Count' (high variance)
• Consider scaling/normalizing 'URG Flag Count' (high variance)
• Consider scaling/normalizing 'ECE Flag Count' (high variance)
• Consider scaling/normalizing 'Avg Fwd Segment Size' (high variance)
• Consider scaling/normalizing 'Fwd Header Length.1' (high variance)
• Consider scaling/normalizing 'Subflow Fwd Packets' (high variance)
• Consider scaling/normalizing 'Subflow Fwd Bytes' (high variance)
• Consider scaling/normalizing 'Subflow Bwd Packets' (high variance)
• Consider scaling/normalizing 'Subflow Bwd Bytes' (high variance)
• Consider scaling/normalizing 'Init_Win_bytes_backward' (high variance)
• Consider scaling/normalizing 'act_data_pkt_fwd' (high variance)
• Consider scaling/normalizing 'Active Mean' (high variance)
• Consider scaling/normalizing 'Active Std' (high variance)
• Consider scaling/normalizing 'Active Max' (high variance)
• Consider scaling/normalizing 'Active Min' (high variance)
• Consider scaling/normalizing 'Idle Mean' (high variance)
• Consider scaling/normalizing 'Idle Std' (high variance)
• Consider scaling/normalizing 'Idle Max' (high variance)
• Consider scaling/normalizing 'Idle Min' (high variance)

================================================================================

# Export summary statistics (uncomment to use)
# if df is not None:
#     # Numerical summary
#     if numeric_cols:
#         df[numeric_cols].describe().to_csv('numerical_summary.csv')
#         print("✓ Numerical summary exported to 'numerical_summary.csv'")
#     
#     # Missing values summary
#     missing_summary = pd.DataFrame({
#         'Column': df.columns,
#         'Missing_Count': df.isnull().sum().values,
#         'Missing_Percentage': (df.isnull().sum().values / len(df) * 100)
#     })
#     missing_summary.to_csv('missing_values_summary.csv', index=False)
#     print("✓ Missing values summary exported to 'missing_values_summary.csv'")
#     
#     # Correlation matrix
#     if len(numeric_cols) >= 2:
#         df[numeric_cols].corr().to_csv('correlation_matrix.csv')
#         print("✓ Correlation matrix exported to 'correlation_matrix.csv'")

# Import additional libraries for clustering
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import DBSCAN, AgglomerativeClustering
from sklearn.metrics import silhouette_score, davies_bouldin_score, calinski_harabasz_score
from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from scipy.cluster.hierarchy import dendrogram, linkage
from scipy.spatial.distance import pdist
import time

print("Clustering libraries imported successfully!")

Clustering libraries imported successfully!

def preprocess_for_clustering(df, label_column='Label', remove_high_corr=True, corr_threshold=0.95):
    """
    Prepare data for clustering analysis.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Input dataset
    label_column : str
        Name of the label column (will be removed from features but saved)
    remove_high_corr : bool
        Whether to remove highly correlated features
    corr_threshold : float
        Correlation threshold for feature removal (0.95 means remove if corr > 0.95)
    
    Returns:
    --------
    X_scaled : numpy array
        Scaled feature matrix ready for clustering
    labels : pandas Series
        True labels (for validation only, not used in clustering)
    feature_names : list
        Names of features after preprocessing
    scaler : StandardScaler
        Fitted scaler (for inverse transform if needed)
    """
    print("=" * 80)
    print("PREPROCESSING FOR CLUSTERING")
    print("=" * 80)
    
    # Save labels for validation (not used in training!)
    labels = df[label_column].copy() if label_column in df.columns else None
    
    # Remove non-feature columns
    non_feature_cols = ['Flow ID', 'Source IP', 'Destination IP', 'Timestamp', label_column]
    df_features = df.drop(columns=[col for col in non_feature_cols if col in df.columns], errors='ignore')
    
    # Select only numeric columns
    df_numeric = df_features.select_dtypes(include=[np.number])
    
    print(f"\n1. Starting features: {df_numeric.shape[1]}")
    print(f"   Starting samples: {df_numeric.shape[0]:,}")
    
    # Handle infinite values
    print("\n2. Handling infinite values...")
    df_clean = df_numeric.replace([np.inf, -np.inf], np.nan)
    inf_count = np.isinf(df_numeric.values).sum()
    print(f"   Replaced {inf_count} infinite values with NaN")
    
    # Handle missing values
    print("\n3. Handling missing values...")
    missing_before = df_clean.isnull().sum().sum()
    df_clean = df_clean.fillna(df_clean.median())
    print(f"   Imputed {missing_before} missing values with column medians")
    
    # Remove highly correlated features
    if remove_high_corr:
        print(f"\n4. Removing highly correlated features (threshold={corr_threshold})...")
        corr_matrix = df_clean.corr().abs()
        upper_triangle = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
        
        # Find features with correlation greater than threshold
        to_drop = [column for column in upper_triangle.columns if any(upper_triangle[column] > corr_threshold)]
        
        print(f"   Removing {len(to_drop)} highly correlated features")
        if len(to_drop) > 0:
            print(f"   First 10 removed: {to_drop[:10]}")
        
        df_clean = df_clean.drop(columns=to_drop)
    
    print(f"\n5. Features after correlation removal: {df_clean.shape[1]}")
    
    # Scale features
    print("\n6. Scaling features...")
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(df_clean)
    
    feature_names = df_clean.columns.tolist()
    
    print("\n✓ Preprocessing complete!")
    print(f"  Final shape: {X_scaled.shape[0]:,} samples × {X_scaled.shape[1]} features")
    print("=" * 80)
    
    return X_scaled, labels, feature_names, scaler

# Preprocess the data
if df is not None:
    X_scaled, true_labels, feature_names, scaler = preprocess_for_clustering(df)
    print(f"\n📊 Ready for clustering with {len(feature_names)} features")

================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 225,745

2. Handling infinite values...
   Replaced 64 infinite values with NaN

3. Handling missing values...
   Imputed 68 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 26 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Fwd Packet Length Std', 'Bwd Packet Length Mean', 'Bwd Packet Length Std', 'Flow IAT Max', 'Fwd IAT Total', 'Fwd IAT Std', 'Fwd IAT Max', 'Bwd IAT Max']

5. Features after correlation removal: 54

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 225,745 samples × 54 features
================================================================================

📊 Ready for clustering with 54 features

def apply_pca(X, n_components=0.95, max_components=20):
    """
    Apply PCA for dimensionality reduction.
    
    Parameters:
    -----------
    X : numpy array
        Scaled feature matrix
    n_components : float or int
        If float (0-1): retain this much variance
        If int: number of components to keep
    max_components : int
        Maximum number of components (cap for variance-based selection)
    
    Returns:
    --------
    X_pca : numpy array
        Reduced feature matrix
    pca : PCA object
        Fitted PCA transformer
    """
    print("=" * 80)
    print("DIMENSIONALITY REDUCTION WITH PCA")
    print("=" * 80)
    
    print(f"\nOriginal dimensions: {X.shape[1]}")
    
    # Apply PCA
    pca = PCA(n_components=n_components)
    X_pca = pca.fit_transform(X)
    
    # Cap at max_components if needed
    if X_pca.shape[1] > max_components:
        print(f"\nCapping at {max_components} components for computational efficiency")
        pca = PCA(n_components=max_components)
        X_pca = pca.fit_transform(X)
    
    n_components_used = X_pca.shape[1]
    variance_explained = pca.explained_variance_ratio_.sum()
    
    print(f"\nReduced to: {n_components_used} components")
    print(f"Variance explained: {variance_explained*100:.2f}%")
    
    # Plot variance explained
    plt.figure(figsize=(12, 4))
    
    plt.subplot(1, 2, 1)
    plt.bar(range(1, n_components_used + 1), pca.explained_variance_ratio_)
    plt.xlabel('Principal Component')
    plt.ylabel('Variance Explained Ratio')
    plt.title('Variance Explained by Each Component')
    plt.grid(True, alpha=0.3)
    
    plt.subplot(1, 2, 2)
    plt.plot(range(1, n_components_used + 1), np.cumsum(pca.explained_variance_ratio_), 'o-')
    plt.xlabel('Number of Components')
    plt.ylabel('Cumulative Variance Explained')
    plt.title('Cumulative Variance Explained')
    plt.grid(True, alpha=0.3)
    plt.axhline(y=0.95, color='r', linestyle='--', label='95% variance')
    plt.legend()
    
    plt.tight_layout()
    plt.show()
    
    print("\n✓ PCA complete!")
    print("=" * 80)
    
    return X_pca, pca

# Apply PCA (uncomment to use)
# Recommended: reduces computational cost and curse of dimensionality
use_pca = True  # Set to False to cluster on all features

if use_pca and X_scaled is not None:
    X_for_clustering, pca_model = apply_pca(X_scaled, n_components=0.95, max_components=20)
else:
    X_for_clustering = X_scaled
    print("Skipping PCA - using all features for clustering")

================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 54

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 93.60%

✓ PCA complete!
================================================================================

def tune_dbscan(X, eps_values=[0.3, 0.5, 0.7, 1.0, 1.5], min_samples_values=[5, 10, 20, 50], 
                sample_size=10000, random_state=42):
    """
    Test multiple DBSCAN parameter combinations.
    Uses sampling for large datasets to speed up tuning.
    
    Parameters:
    -----------
    X : numpy array
        Feature matrix
    eps_values : list
        List of eps values to test
    min_samples_values : list
        List of min_samples values to test
    sample_size : int
        Number of samples to use for tuning (speeds up for large datasets)
    
    Returns:
    --------
    results_df : pandas DataFrame
        Results for each parameter combination
    """
    print("=" * 80)
    print("DBSCAN PARAMETER TUNING")
    print("=" * 80)
    
    # Sample data if too large
    if X.shape[0] > sample_size:
        print(f"\nSampling {sample_size:,} points from {X.shape[0]:,} for faster tuning...")
        np.random.seed(random_state)
        indices = np.random.choice(X.shape[0], sample_size, replace=False)
        X_sample = X[indices]
    else:
        X_sample = X
    
    results = []
    
    print(f"\nTesting {len(eps_values)} eps values × {len(min_samples_values)} min_samples values...")
    print("This may take a few minutes...\n")
    
    for eps in eps_values:
        for min_samples in min_samples_values:
            start_time = time.time()
            
            # Fit DBSCAN
            dbscan = DBSCAN(eps=eps, min_samples=min_samples, n_jobs=-1)
            labels = dbscan.fit_predict(X_sample)
            
            # Calculate metrics
            n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
            n_noise = list(labels).count(-1)
            noise_pct = (n_noise / len(labels)) * 100
            
            # Silhouette score (only if we have valid clusters)
            silhouette = np.nan
            if n_clusters > 1 and n_noise < len(labels) - n_clusters:
                # Only compute on non-noise points
                mask = labels != -1
                if mask.sum() > n_clusters:
                    try:
                        silhouette = silhouette_score(X_sample[mask], labels[mask])
                    except:
                        silhouette = np.nan
            
            elapsed = time.time() - start_time
            
            results.append({
                'eps': eps,
                'min_samples': min_samples,
                'n_clusters': n_clusters,
                'n_noise': n_noise,
                'noise_pct': noise_pct,
                'silhouette': silhouette,
                'time_sec': elapsed
            })
            
            # Format silhouette score properly
            silhouette_str = f"{silhouette:.3f}" if not np.isnan(silhouette) else "N/A"
            print(f"eps={eps}, min_samples={min_samples}: "
                  f"{n_clusters} clusters, {noise_pct:.1f}% noise, "
                  f"silhouette={silhouette_str}")
    
    results_df = pd.DataFrame(results)
    
    print("\n" + "=" * 80)
    print("TUNING RESULTS SUMMARY")
    print("=" * 80)
    
    # Show best configurations by silhouette score
    valid_results = results_df.dropna(subset=['silhouette'])
    if len(valid_results) > 0:
        print("\nTop 5 configurations by silhouette score:")
        top_5 = valid_results.nlargest(5, 'silhouette')[['eps', 'min_samples', 'n_clusters', 'noise_pct', 'silhouette']]
        display(top_5)
    
    # Show all results
    print("\nAll parameter combinations:")
    display(results_df)
    
    return results_df

# Run parameter tuning
if X_for_clustering is not None:
    dbscan_results = tune_dbscan(X_for_clustering)

================================================================================
DBSCAN PARAMETER TUNING
================================================================================

Sampling 10,000 points from 225,745 for faster tuning...

Testing 5 eps values × 4 min_samples values...
This may take a few minutes...

eps=0.3, min_samples=5: 98 clusters, 19.4% noise, silhouette=0.259
eps=0.3, min_samples=10: 53 clusters, 24.4% noise, silhouette=0.337
eps=0.3, min_samples=20: 32 clusters, 32.8% noise, silhouette=0.435
eps=0.3, min_samples=50: 14 clusters, 45.2% noise, silhouette=0.589
eps=0.5, min_samples=5: 78 clusters, 12.7% noise, silhouette=0.500
eps=0.5, min_samples=10: 41 clusters, 17.3% noise, silhouette=0.530
eps=0.5, min_samples=20: 19 clusters, 22.7% noise, silhouette=0.572
eps=0.5, min_samples=50: 8 clusters, 31.4% noise, silhouette=0.689
eps=0.7, min_samples=5: 69 clusters, 8.6% noise, silhouette=0.528
eps=0.7, min_samples=10: 41 clusters, 12.1% noise, silhouette=0.543
eps=0.7, min_samples=20: 25 clusters, 17.1% noise, silhouette=0.550
eps=0.7, min_samples=50: 12 clusters, 23.7% noise, silhouette=0.612
eps=1.0, min_samples=5: 63 clusters, 5.6% noise, silhouette=0.549
eps=1.0, min_samples=10: 39 clusters, 8.2% noise, silhouette=0.572
eps=1.0, min_samples=20: 20 clusters, 12.0% noise, silhouette=0.592
eps=1.0, min_samples=50: 13 clusters, 19.0% noise, silhouette=0.698
eps=1.5, min_samples=5: 45 clusters, 3.6% noise, silhouette=0.465
eps=1.5, min_samples=10: 30 clusters, 5.0% noise, silhouette=0.477
eps=1.5, min_samples=20: 19 clusters, 8.0% noise, silhouette=0.509
eps=1.5, min_samples=50: 12 clusters, 12.3% noise, silhouette=0.580

================================================================================
TUNING RESULTS SUMMARY
================================================================================

Top 5 configurations by silhouette score:

All parameter combinations:

def apply_dbscan(X, eps=0.5, min_samples=10, sample_for_viz=5000, random_state=42):
    """
    Apply DBSCAN clustering with chosen parameters.
    
    Parameters:
    -----------
    X : numpy array
        Feature matrix
    eps : float
        DBSCAN eps parameter
    min_samples : int
        DBSCAN min_samples parameter
    sample_for_viz : int
        Number of samples to use for visualization (for large datasets)
    
    Returns:
    --------
    cluster_labels : numpy array
        Cluster assignments (-1 for noise)
    dbscan_model : DBSCAN
        Fitted DBSCAN model
    """
    print("=" * 80)
    print("APPLYING DBSCAN CLUSTERING")
    print("=" * 80)
    
    print(f"\nParameters:")
    print(f"  eps = {eps}")
    print(f"  min_samples = {min_samples}")
    print(f"\nClustering {X.shape[0]:,} samples with {X.shape[1]} features...")
    
    start_time = time.time()
    
    # Fit DBSCAN
    dbscan = DBSCAN(eps=eps, min_samples=min_samples, n_jobs=-1)
    cluster_labels = dbscan.fit_predict(X)
    
    elapsed = time.time() - start_time
    
    # Analyze results
    n_clusters = len(set(cluster_labels)) - (1 if -1 in cluster_labels else 0)
    n_noise = list(cluster_labels).count(-1)
    
    print(f"\n✓ Clustering complete in {elapsed:.2f} seconds")
    print(f"\nResults:")
    print(f"  Clusters found: {n_clusters}")
    print(f"  Noise points: {n_noise:,} ({n_noise/len(cluster_labels)*100:.1f}%)")
    print(f"  Clustered points: {len(cluster_labels) - n_noise:,} ({(len(cluster_labels)-n_noise)/len(cluster_labels)*100:.1f}%)")
    
    # Cluster size distribution
    print("\nCluster size distribution:")
    unique, counts = np.unique(cluster_labels[cluster_labels != -1], return_counts=True)
    for cluster_id, count in zip(unique, counts):
        print(f"  Cluster {cluster_id}: {count:,} points ({count/len(cluster_labels)*100:.1f}%)")
    
    # Visualize (use PCA if not already reduced to 2D)
    print("\nGenerating visualization...")
    
    # Sample for visualization if dataset is large
    if X.shape[0] > sample_for_viz:
        np.random.seed(random_state)
        viz_indices = np.random.choice(X.shape[0], sample_for_viz, replace=False)
        X_viz = X[viz_indices]
        labels_viz = cluster_labels[viz_indices]
    else:
        X_viz = X
        labels_viz = cluster_labels
    
    # Reduce to 2D for visualization if needed
    if X_viz.shape[1] > 2:
        pca_viz = PCA(n_components=2)
        X_2d = pca_viz.fit_transform(X_viz)
        var_explained = pca_viz.explained_variance_ratio_.sum()
        title_suffix = f"(2D PCA projection, {var_explained*100:.1f}% variance)"
    else:
        X_2d = X_viz
        title_suffix = ""
    
    # Plot
    plt.figure(figsize=(14, 6))
    
    # Plot 1: All clusters
    plt.subplot(1, 2, 1)
    scatter = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=labels_viz, cmap='tab10', 
                         alpha=0.6, s=10, edgecolors='none')
    plt.colorbar(scatter, label='Cluster')
    plt.xlabel('First Component')
    plt.ylabel('Second Component')
    plt.title(f'DBSCAN Clustering Results\n{title_suffix}')
    plt.grid(True, alpha=0.3)
    
    # Plot 2: Highlight noise points
    plt.subplot(1, 2, 2)
    noise_mask = labels_viz == -1
    plt.scatter(X_2d[~noise_mask, 0], X_2d[~noise_mask, 1], 
               c=labels_viz[~noise_mask], cmap='tab10', alpha=0.5, s=10, label='Clusters')
    plt.scatter(X_2d[noise_mask, 0], X_2d[noise_mask, 1], 
               c='red', marker='x', s=20, alpha=0.7, label='Noise')
    plt.xlabel('First Component')
    plt.ylabel('Second Component')
    plt.title('Noise Points Highlighted')
    plt.legend()
    plt.grid(True, alpha=0.3)
    
    plt.tight_layout()
    plt.show()
    
    print("\n" + "=" * 80)
    
    return cluster_labels, dbscan

# Apply DBSCAN with chosen parameters
# Based on tuning results, choose appropriate eps and min_samples
if X_for_clustering is not None:
    # ADJUST THESE PARAMETERS based on your tuning results above!
    chosen_eps = 0.5
    chosen_min_samples = 10
    
    dbscan_labels, dbscan_model = apply_dbscan(X_for_clustering, 
                                                eps=chosen_eps, 
                                                min_samples=chosen_min_samples)

================================================================================
APPLYING DBSCAN CLUSTERING
================================================================================

Parameters:
  eps = 0.5
  min_samples = 10

Clustering 225,745 samples with 20 features...

✓ Clustering complete in 13.91 seconds

Results:
  Clusters found: 294
  Noise points: 9,481 (4.2%)
  Clustered points: 216,264 (95.8%)

Cluster size distribution:
  Cluster 0: 414 points (0.2%)
  Cluster 1: 1,619 points (0.7%)
  Cluster 2: 50,094 points (22.2%)
  Cluster 3: 174 points (0.1%)
  Cluster 4: 23 points (0.0%)
  Cluster 5: 61 points (0.0%)
  Cluster 6: 81 points (0.0%)
  Cluster 7: 139 points (0.1%)
  Cluster 8: 35 points (0.0%)
  Cluster 9: 1,191 points (0.5%)
  Cluster 10: 38 points (0.0%)
  Cluster 11: 12 points (0.0%)
  Cluster 12: 12 points (0.0%)
  Cluster 13: 13 points (0.0%)
  Cluster 14: 241 points (0.1%)
  Cluster 15: 35 points (0.0%)
  Cluster 16: 128 points (0.1%)
  Cluster 17: 438 points (0.2%)
  Cluster 18: 58 points (0.0%)
  Cluster 19: 67 points (0.0%)
  Cluster 20: 20 points (0.0%)
  Cluster 21: 16,457 points (7.3%)
  Cluster 22: 1,226 points (0.5%)
  Cluster 23: 31 points (0.0%)
  Cluster 24: 24 points (0.0%)
  Cluster 25: 27 points (0.0%)
  Cluster 26: 233 points (0.1%)
  Cluster 27: 460 points (0.2%)
  Cluster 28: 829 points (0.4%)
  Cluster 29: 432 points (0.2%)
  Cluster 30: 12,799 points (5.7%)
  Cluster 31: 123 points (0.1%)
  Cluster 32: 514 points (0.2%)
  Cluster 33: 70 points (0.0%)
  Cluster 34: 259 points (0.1%)
  Cluster 35: 20 points (0.0%)
  Cluster 36: 20 points (0.0%)
  Cluster 37: 42 points (0.0%)
  Cluster 38: 2,234 points (1.0%)
  Cluster 39: 32 points (0.0%)
  Cluster 40: 10 points (0.0%)
  Cluster 41: 138 points (0.1%)
  Cluster 42: 376 points (0.2%)
  Cluster 43: 42 points (0.0%)
  Cluster 44: 23 points (0.0%)
  Cluster 45: 13 points (0.0%)
  Cluster 46: 21 points (0.0%)
  Cluster 47: 345 points (0.2%)
  Cluster 48: 235 points (0.1%)
  Cluster 49: 11 points (0.0%)
  Cluster 50: 603 points (0.3%)
  Cluster 51: 54,241 points (24.0%)
  Cluster 52: 237 points (0.1%)
  Cluster 53: 16 points (0.0%)
  Cluster 54: 116 points (0.1%)
  Cluster 55: 65 points (0.0%)
  Cluster 56: 328 points (0.1%)
  Cluster 57: 15 points (0.0%)
  Cluster 58: 37 points (0.0%)
  Cluster 59: 50 points (0.0%)
  Cluster 60: 11 points (0.0%)
  Cluster 61: 105 points (0.0%)
  Cluster 62: 60 points (0.0%)
  Cluster 63: 89 points (0.0%)
  Cluster 64: 10 points (0.0%)
  Cluster 65: 760 points (0.3%)
  Cluster 66: 227 points (0.1%)
  Cluster 67: 51 points (0.0%)
  Cluster 68: 316 points (0.1%)
  Cluster 69: 15 points (0.0%)
  Cluster 70: 178 points (0.1%)
  Cluster 71: 72 points (0.0%)
  Cluster 72: 55 points (0.0%)
  Cluster 73: 21 points (0.0%)
  Cluster 74: 80 points (0.0%)
  Cluster 75: 40 points (0.0%)
  Cluster 76: 10 points (0.0%)
  Cluster 77: 147 points (0.1%)
  Cluster 78: 10 points (0.0%)
  Cluster 79: 14 points (0.0%)
  Cluster 80: 67 points (0.0%)
  Cluster 81: 10 points (0.0%)
  Cluster 82: 16 points (0.0%)
  Cluster 83: 10 points (0.0%)
  Cluster 84: 20 points (0.0%)
  Cluster 85: 15 points (0.0%)
  Cluster 86: 119 points (0.1%)
  Cluster 87: 162 points (0.1%)
  Cluster 88: 30 points (0.0%)
  Cluster 89: 33 points (0.0%)
  Cluster 90: 17 points (0.0%)
  Cluster 91: 23 points (0.0%)
  Cluster 92: 110 points (0.0%)
  Cluster 93: 42 points (0.0%)
  Cluster 94: 81 points (0.0%)
  Cluster 95: 158 points (0.1%)
  Cluster 96: 20 points (0.0%)
  Cluster 97: 29 points (0.0%)
  Cluster 98: 75 points (0.0%)
  Cluster 99: 11 points (0.0%)
  Cluster 100: 17 points (0.0%)
  Cluster 101: 16 points (0.0%)
  Cluster 102: 31 points (0.0%)
  Cluster 103: 18 points (0.0%)
  Cluster 104: 13 points (0.0%)
  Cluster 105: 30 points (0.0%)
  Cluster 106: 20 points (0.0%)
  Cluster 107: 12 points (0.0%)
  Cluster 108: 194 points (0.1%)
  Cluster 109: 10 points (0.0%)
  Cluster 110: 57 points (0.0%)
  Cluster 111: 69 points (0.0%)
  Cluster 112: 31 points (0.0%)
  Cluster 113: 26 points (0.0%)
  Cluster 114: 13 points (0.0%)
  Cluster 115: 29 points (0.0%)
  Cluster 116: 70 points (0.0%)
  Cluster 117: 18 points (0.0%)
  Cluster 118: 30 points (0.0%)
  Cluster 119: 28 points (0.0%)
  Cluster 120: 47 points (0.0%)
  Cluster 121: 28 points (0.0%)
  Cluster 122: 15 points (0.0%)
  Cluster 123: 97 points (0.0%)
  Cluster 124: 31 points (0.0%)
  Cluster 125: 11 points (0.0%)
  Cluster 126: 15 points (0.0%)
  Cluster 127: 59 points (0.0%)
  Cluster 128: 46 points (0.0%)
  Cluster 129: 31 points (0.0%)
  Cluster 130: 11 points (0.0%)
  Cluster 131: 17 points (0.0%)
  Cluster 132: 36 points (0.0%)
  Cluster 133: 80 points (0.0%)
  Cluster 134: 55 points (0.0%)
  Cluster 135: 27 points (0.0%)
  Cluster 136: 318 points (0.1%)
  Cluster 137: 15 points (0.0%)
  Cluster 138: 18 points (0.0%)
  Cluster 139: 56 points (0.0%)
  Cluster 140: 62 points (0.0%)
  Cluster 141: 210 points (0.1%)
  Cluster 142: 27 points (0.0%)
  Cluster 143: 105 points (0.0%)
  Cluster 144: 63 points (0.0%)
  Cluster 145: 26 points (0.0%)
  Cluster 146: 91 points (0.0%)
  Cluster 147: 20 points (0.0%)
  Cluster 148: 10 points (0.0%)
  Cluster 149: 22 points (0.0%)
  Cluster 150: 41 points (0.0%)
  Cluster 151: 29 points (0.0%)
  Cluster 152: 28 points (0.0%)
  Cluster 153: 10 points (0.0%)
  Cluster 154: 21 points (0.0%)
  Cluster 155: 18 points (0.0%)
  Cluster 156: 17 points (0.0%)
  Cluster 157: 13 points (0.0%)
  Cluster 158: 18 points (0.0%)
  Cluster 159: 46 points (0.0%)
  Cluster 160: 28 points (0.0%)
  Cluster 161: 20 points (0.0%)
  Cluster 162: 101 points (0.0%)
  Cluster 163: 22 points (0.0%)
  Cluster 164: 16 points (0.0%)
  Cluster 165: 14 points (0.0%)
  Cluster 166: 15 points (0.0%)
  Cluster 167: 22 points (0.0%)
  Cluster 168: 15 points (0.0%)
  Cluster 169: 40 points (0.0%)
  Cluster 170: 13 points (0.0%)
  Cluster 171: 17 points (0.0%)
  Cluster 172: 22 points (0.0%)
  Cluster 173: 22 points (0.0%)
  Cluster 174: 12 points (0.0%)
  Cluster 175: 22 points (0.0%)
  Cluster 176: 20 points (0.0%)
  Cluster 177: 29 points (0.0%)
  Cluster 178: 14 points (0.0%)
  Cluster 179: 30 points (0.0%)
  Cluster 180: 25 points (0.0%)
  Cluster 181: 26 points (0.0%)
  Cluster 182: 43 points (0.0%)
  Cluster 183: 17 points (0.0%)
  Cluster 184: 34 points (0.0%)
  Cluster 185: 561 points (0.2%)
  Cluster 186: 10 points (0.0%)
  Cluster 187: 130 points (0.1%)
  Cluster 188: 20 points (0.0%)
  Cluster 189: 10 points (0.0%)
  Cluster 190: 10 points (0.0%)
  Cluster 191: 659 points (0.3%)
  Cluster 192: 14 points (0.0%)
  Cluster 193: 10 points (0.0%)
  Cluster 194: 11 points (0.0%)
  Cluster 195: 15 points (0.0%)
  Cluster 196: 35 points (0.0%)
  Cluster 197: 19 points (0.0%)
  Cluster 198: 16 points (0.0%)
  Cluster 199: 21 points (0.0%)
  Cluster 200: 31 points (0.0%)
  Cluster 201: 145 points (0.1%)
  Cluster 202: 71 points (0.0%)
  Cluster 203: 18 points (0.0%)
  Cluster 204: 10 points (0.0%)
  Cluster 205: 10,749 points (4.8%)
  Cluster 206: 1,241 points (0.5%)
  Cluster 207: 6,266 points (2.8%)
  Cluster 208: 8,450 points (3.7%)
  Cluster 209: 967 points (0.4%)
  Cluster 210: 524 points (0.2%)
  Cluster 211: 3,687 points (1.6%)
  Cluster 212: 425 points (0.2%)
  Cluster 213: 13,395 points (5.9%)
  Cluster 214: 79 points (0.0%)
  Cluster 215: 1,955 points (0.9%)
  Cluster 216: 407 points (0.2%)
  Cluster 217: 3,085 points (1.4%)
  Cluster 218: 15 points (0.0%)
  Cluster 219: 10 points (0.0%)
  Cluster 220: 10 points (0.0%)
  Cluster 221: 13 points (0.0%)
  Cluster 222: 119 points (0.1%)
  Cluster 223: 13 points (0.0%)
  Cluster 224: 316 points (0.1%)
  Cluster 225: 1,709 points (0.8%)
  Cluster 226: 14 points (0.0%)
  Cluster 227: 18 points (0.0%)
  Cluster 228: 78 points (0.0%)
  Cluster 229: 372 points (0.2%)
  Cluster 230: 13 points (0.0%)
  Cluster 231: 10 points (0.0%)
  Cluster 232: 143 points (0.1%)
  Cluster 233: 25 points (0.0%)
  Cluster 234: 4,223 points (1.9%)
  Cluster 235: 21 points (0.0%)
  Cluster 236: 40 points (0.0%)
  Cluster 237: 9 points (0.0%)
  Cluster 238: 12 points (0.0%)
  Cluster 239: 36 points (0.0%)
  Cluster 240: 14 points (0.0%)
  Cluster 241: 12 points (0.0%)
  Cluster 242: 40 points (0.0%)
  Cluster 243: 19 points (0.0%)
  Cluster 244: 35 points (0.0%)
  Cluster 245: 16 points (0.0%)
  Cluster 246: 27 points (0.0%)
  Cluster 247: 13 points (0.0%)
  Cluster 248: 30 points (0.0%)
  Cluster 249: 88 points (0.0%)
  Cluster 250: 506 points (0.2%)
  Cluster 251: 18 points (0.0%)
  Cluster 252: 11 points (0.0%)
  Cluster 253: 11 points (0.0%)
  Cluster 254: 7 points (0.0%)
  Cluster 255: 17 points (0.0%)
  Cluster 256: 5 points (0.0%)
  Cluster 257: 12 points (0.0%)
  Cluster 258: 11 points (0.0%)
  Cluster 259: 11 points (0.0%)
  Cluster 260: 10 points (0.0%)
  Cluster 261: 23 points (0.0%)
  Cluster 262: 33 points (0.0%)
  Cluster 263: 22 points (0.0%)
  Cluster 264: 10 points (0.0%)
  Cluster 265: 30 points (0.0%)
  Cluster 266: 12 points (0.0%)
  Cluster 267: 14 points (0.0%)
  Cluster 268: 11 points (0.0%)
  Cluster 269: 10 points (0.0%)
  Cluster 270: 13 points (0.0%)
  Cluster 271: 14 points (0.0%)
  Cluster 272: 27 points (0.0%)
  Cluster 273: 15 points (0.0%)
  Cluster 274: 14 points (0.0%)
  Cluster 275: 22 points (0.0%)
  Cluster 276: 14 points (0.0%)
  Cluster 277: 11 points (0.0%)
  Cluster 278: 36 points (0.0%)
  Cluster 279: 292 points (0.1%)
  Cluster 280: 10 points (0.0%)
  Cluster 281: 32 points (0.0%)
  Cluster 282: 12 points (0.0%)
  Cluster 283: 7 points (0.0%)
  Cluster 284: 11 points (0.0%)
  Cluster 285: 13 points (0.0%)
  Cluster 286: 15 points (0.0%)
  Cluster 287: 17 points (0.0%)
  Cluster 288: 10 points (0.0%)
  Cluster 289: 43 points (0.0%)
  Cluster 290: 10 points (0.0%)
  Cluster 291: 19 points (0.0%)
  Cluster 292: 26 points (0.0%)
  Cluster 293: 11 points (0.0%)

Generating visualization...

================================================================================

def visualize_dendrogram(X, method='ward', sample_size=1000, random_state=42):
    """
    Create and visualize a dendrogram for hierarchical clustering.
    Uses sampling for large datasets.
    
    Parameters:
    -----------
    X : numpy array
        Feature matrix
    method : str
        Linkage method ('ward', 'complete', 'average', 'single')
    sample_size : int
        Number of samples to use for dendrogram (dendrograms are slow for large datasets)
    """
    print("=" * 80)
    print("HIERARCHICAL CLUSTERING - DENDROGRAM")
    print("=" * 80)
    
    # Sample for dendrogram visualization (dendrograms are very slow for large datasets)
    if X.shape[0] > sample_size:
        print(f"\nSampling {sample_size:,} points from {X.shape[0]:,} for dendrogram...")
        np.random.seed(random_state)
        indices = np.random.choice(X.shape[0], sample_size, replace=False)
        X_sample = X[indices]
    else:
        X_sample = X
    
    print(f"\nComputing linkage matrix using '{method}' method...")
    print("This may take a few minutes...")
    
    start_time = time.time()
    linkage_matrix = linkage(X_sample, method=method)
    elapsed = time.time() - start_time
    
    print(f"✓ Linkage computed in {elapsed:.2f} seconds")
    
    # Plot dendrogram
    plt.figure(figsize=(15, 7))
    
    dendrogram(
        linkage_matrix,
        truncate_mode='lastp',  # Show only the last p merged clusters
        p=30,  # Show last 30 merges
        leaf_rotation=90,
        leaf_font_size=10,
        show_contracted=True,
        color_threshold=None
    )
    
    plt.title(f'Hierarchical Clustering Dendrogram (method={method})\n'
              f'Showing last 30 merges out of {len(X_sample)} samples', fontsize=14, fontweight='bold')
    plt.xlabel('Sample Index or (Cluster Size)', fontsize=12)
    plt.ylabel('Distance', fontsize=12)
    plt.axhline(y=10, color='r', linestyle='--', linewidth=1, label='Potential cut height')
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()
    
    print("\n💡 Interpretation:")
    print("   - Vertical axis: Distance between clusters when merged")
    print("   - Horizontal axis: Samples or cluster groups")
    print("   - Cut the dendrogram horizontally at different heights to get different numbers of clusters")
    print("   - Large vertical distances suggest good separation between clusters")
    
    print("\n" + "=" * 80)
    
    return linkage_matrix

# Visualize dendrogram
if X_for_clustering is not None:
    linkage_matrix = visualize_dendrogram(X_for_clustering, method='ward')

================================================================================
HIERARCHICAL CLUSTERING - DENDROGRAM
================================================================================

Sampling 1,000 points from 225,745 for dendrogram...

Computing linkage matrix using 'ward' method...
This may take a few minutes...
✓ Linkage computed in 0.01 seconds

💡 Interpretation:
   - Vertical axis: Distance between clusters when merged
   - Horizontal axis: Samples or cluster groups
   - Cut the dendrogram horizontally at different heights to get different numbers of clusters
   - Large vertical distances suggest good separation between clusters

================================================================================

def apply_hierarchical_clustering(X, n_clusters=None, distance_threshold=None, 
                                 linkage='ward', max_samples=10000, 
                                 sample_for_viz=5000, random_state=42):
    """
    Apply hierarchical clustering.
    
    IMPORTANT: Hierarchical clustering has O(n²) memory and O(n³) time complexity.
    For large datasets, this function automatically samples to prevent crashes.
    
    Parameters:
    -----------
    X : numpy array
        Feature matrix
    n_clusters : int or None
        Number of clusters to find (if None, use distance_threshold)
    distance_threshold : float or None
        Distance threshold for cutting dendrogram (if None, use n_clusters)
    linkage : str
        Linkage method
    max_samples : int
        Maximum number of samples to use for clustering (prevents memory issues)
    sample_for_viz : int
        Number of samples for visualization
    
    Returns:
    --------
    cluster_labels : numpy array
        Cluster assignments (only for sampled data if dataset is large!)
    hierarchical_model : AgglomerativeClustering
        Fitted hierarchical clustering model
    sample_indices : numpy array or None
        Indices of sampled data (None if no sampling)
    """
    print("=" * 80)
    print("APPLYING HIERARCHICAL CLUSTERING")
    print("=" * 80)
    
    if n_clusters is None and distance_threshold is None:
        print("\n⚠️ Must specify either n_clusters or distance_threshold")
        print("   Using n_clusters=5 as default")
        n_clusters = 5
    
    # Check if dataset is too large for hierarchical clustering
    if X.shape[0] > max_samples:
        print(f"\n⚠️  IMPORTANT: Dataset has {X.shape[0]:,} samples")
        print(f"   Hierarchical clustering is computationally expensive for large datasets")
        print(f"   Using random sample of {max_samples:,} samples to prevent memory issues\n")
        
        np.random.seed(random_state)
        sample_indices = np.random.choice(X.shape[0], max_samples, replace=False)
        X_cluster = X[sample_indices]
        is_sampled = True
    else:
        X_cluster = X
        sample_indices = None
        is_sampled = False
    
    print(f"Parameters:")
    print(f"  linkage = {linkage}")
    if n_clusters is not None:
        print(f"  n_clusters = {n_clusters}")
    if distance_threshold is not None:
        print(f"  distance_threshold = {distance_threshold}")
    
    print(f"\nClustering {X_cluster.shape[0]:,} samples with {X_cluster.shape[1]} features...")
    if is_sampled:
        print(f"(Sampled from original {X.shape[0]:,} samples)")
    
    start_time = time.time()
    
    # Fit Agglomerative Clustering
    hierarchical = AgglomerativeClustering(
        n_clusters=n_clusters,
        linkage=linkage,
        distance_threshold=distance_threshold
    )
    cluster_labels = hierarchical.fit_predict(X_cluster)
    
    elapsed = time.time() - start_time
    
    # Analyze results
    n_clusters_found = len(set(cluster_labels))
    
    print(f"\n✓ Clustering complete in {elapsed:.2f} seconds")
    print(f"\nResults:")
    print(f"  Clusters found: {n_clusters_found}")
    
    if is_sampled:
        print(f"\n⚠️  NOTE: Results are based on {max_samples:,} sampled points")
        print(f"   To apply to full dataset, you would need to:")
        print(f"   1. Use the cluster centers from sampled data")
        print(f"   2. Assign remaining points to nearest cluster")
        print(f"   3. Or use a different algorithm (DBSCAN scales better)")
    
    # Cluster size distribution
    print("\nCluster size distribution (in sample):")
    unique, counts = np.unique(cluster_labels, return_counts=True)
    for cluster_id, count in sorted(zip(unique, counts), key=lambda x: x[1], reverse=True):
        print(f"  Cluster {cluster_id}: {count:,} points ({count/len(cluster_labels)*100:.1f}%)")
    
    # Visualize
    print("\nGenerating visualization...")
    
    # Sample for visualization if needed
    if X_cluster.shape[0] > sample_for_viz:
        np.random.seed(random_state)
        viz_indices = np.random.choice(X_cluster.shape[0], sample_for_viz, replace=False)
        X_viz = X_cluster[viz_indices]
        labels_viz = cluster_labels[viz_indices]
    else:
        X_viz = X_cluster
        labels_viz = cluster_labels
    
    # Reduce to 2D for visualization if needed
    if X_viz.shape[1] > 2:
        pca_viz = PCA(n_components=2)
        X_2d = pca_viz.fit_transform(X_viz)
        var_explained = pca_viz.explained_variance_ratio_.sum()
        title_suffix = f"(2D PCA projection, {var_explained*100:.1f}% variance)"
    else:
        X_2d = X_viz
        title_suffix = ""
    
    # Plot
    plt.figure(figsize=(14, 6))
    
    # Plot 1: Cluster visualization
    plt.subplot(1, 2, 1)
    scatter = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=labels_viz, cmap='tab10', 
                         alpha=0.6, s=10, edgecolors='none')
    plt.colorbar(scatter, label='Cluster')
    plt.xlabel('First Component')
    plt.ylabel('Second Component')
    title = f'Hierarchical Clustering Results\n{title_suffix}'
    if is_sampled:
        title += f'\nBased on {max_samples:,} sampled points'
    plt.title(title)
    plt.grid(True, alpha=0.3)
    
    # Plot 2: Cluster size distribution
    plt.subplot(1, 2, 2)
    cluster_sizes = pd.Series(cluster_labels).value_counts().sort_index()
    plt.bar(cluster_sizes.index, cluster_sizes.values)
    plt.xlabel('Cluster ID')
    plt.ylabel('Number of Samples')
    plt.title('Cluster Size Distribution')
    plt.grid(True, alpha=0.3, axis='y')
    
    plt.tight_layout()
    plt.show()
    
    print("\n" + "=" * 80)
    
    return cluster_labels, hierarchical, sample_indices

# Apply hierarchical clustering
if X_for_clustering is not None:
    # You can specify either n_clusters OR distance_threshold
    # Based on dendrogram, choose an appropriate value
    
    # Hierarchical clustering with automatic sampling for large datasets
    hierarchical_labels, hierarchical_model, sample_indices = apply_hierarchical_clustering(
        X_for_clustering, 
        n_clusters=5,  # ADJUST THIS based on dendrogram
        linkage='ward',
        max_samples=10000  # Maximum samples to prevent memory issues
    )
    
    print("\n💡 TIP: For this large dataset, consider using DBSCAN as your primary method.")
    print("   Hierarchical clustering results are based on a sample and may not")
    print("   represent the full dataset accurately.")

================================================================================
APPLYING HIERARCHICAL CLUSTERING
================================================================================

⚠️  IMPORTANT: Dataset has 225,745 samples
   Hierarchical clustering is computationally expensive for large datasets
   Using random sample of 10,000 samples to prevent memory issues

Parameters:
  linkage = ward
  n_clusters = 5

Clustering 10,000 samples with 20 features...
(Sampled from original 225,745 samples)

✓ Clustering complete in 1.44 seconds

Results:
  Clusters found: 5

⚠️  NOTE: Results are based on 10,000 sampled points
   To apply to full dataset, you would need to:
   1. Use the cluster centers from sampled data
   2. Assign remaining points to nearest cluster
   3. Or use a different algorithm (DBSCAN scales better)

Cluster size distribution (in sample):
  Cluster 2: 3,767 points (37.7%)
  Cluster 1: 3,448 points (34.5%)
  Cluster 4: 1,406 points (14.1%)
  Cluster 3: 965 points (9.7%)
  Cluster 0: 414 points (4.1%)

Generating visualization...

================================================================================

💡 TIP: For this large dataset, consider using DBSCAN as your primary method.
   Hierarchical clustering results are based on a sample and may not
   represent the full dataset accurately.

def evaluate_clustering(X, labels, true_labels=None, method_name="Clustering"):
    """
    Evaluate clustering results with multiple metrics.
    
    Parameters:
    -----------
    X : numpy array
        Feature matrix
    labels : numpy array
        Cluster assignments
    true_labels : array-like or None
        Ground truth labels (for validation only)
    method_name : str
        Name of the clustering method (for display)
    
    Returns:
    --------
    metrics_dict : dict
        Dictionary of metric scores
    """
    print("=" * 80)
    print(f"EVALUATING {method_name.upper()}")
    print("=" * 80)
    
    metrics = {}
    
    # Get number of clusters (excluding noise for DBSCAN)
    unique_labels = set(labels)
    n_clusters = len(unique_labels) - (1 if -1 in unique_labels else 0)
    n_noise = list(labels).count(-1) if -1 in labels else 0
    
    print(f"\nClusters: {n_clusters}")
    if n_noise > 0:
        print(f"Noise points: {n_noise} ({n_noise/len(labels)*100:.1f}%)")
    
    # Internal metrics (only if we have enough clusters and non-noise points)
    print("\n" + "-" * 80)
    print("INTERNAL METRICS (no labels used)")
    print("-" * 80)
    
    if n_clusters > 1:
        # Filter out noise points for internal metrics
        mask = labels != -1
        X_filtered = X[mask]
        labels_filtered = labels[mask]
        
        if len(X_filtered) > n_clusters and len(set(labels_filtered)) > 1:
            try:
                # Silhouette Score
                silhouette = silhouette_score(X_filtered, labels_filtered)
                metrics['silhouette'] = silhouette
                print(f"Silhouette Score: {silhouette:.4f}")
                print(f"  Interpretation: {silhouette:.4f} ", end="")
                if silhouette > 0.5:
                    print("(Excellent - strong cluster structure)")
                elif silhouette > 0.3:
                    print("(Good - reasonable cluster structure)")
                elif silhouette > 0:
                    print("(Fair - weak cluster structure)")
                else:
                    print("(Poor - points may be misclassified)")
                
                # Davies-Bouldin Index
                davies_bouldin = davies_bouldin_score(X_filtered, labels_filtered)
                metrics['davies_bouldin'] = davies_bouldin
                print(f"\nDavies-Bouldin Index: {davies_bouldin:.4f}")
                print(f"  Interpretation: {davies_bouldin:.4f} (lower is better)")
                
                # Calinski-Harabasz Index
                calinski = calinski_harabasz_score(X_filtered, labels_filtered)
                metrics['calinski_harabasz'] = calinski
                print(f"\nCalinski-Harabasz Index: {calinski:.2f}")
                print(f"  Interpretation: {calinski:.2f} (higher is better)")
                
            except Exception as e:
                print(f"Could not compute internal metrics: {e}")
        else:
            print("Not enough valid points for internal metrics")
    else:
        print("Need at least 2 clusters for internal metrics")
    
    # External metrics (validation against true labels)
    if true_labels is not None:
        print("\n" + "-" * 80)
        print("EXTERNAL METRICS (validation against true labels)")
        print("⚠️  These are for VALIDATION only - labels were NOT used in clustering!")
        print("-" * 80)
        
        # Filter to same indices (in case of noise points)
        valid_mask = labels != -1 if -1 in labels else np.ones(len(labels), dtype=bool)
        
        if valid_mask.sum() > 0:
            try:
                # Adjusted Rand Index
                ari = adjusted_rand_score(true_labels[valid_mask], labels[valid_mask])
                metrics['adjusted_rand_index'] = ari
                print(f"\nAdjusted Rand Index: {ari:.4f}")
                print(f"  Interpretation: {ari:.4f} ", end="")
                if ari > 0.9:
                    print("(Excellent agreement with true labels)")
                elif ari > 0.7:
                    print("(Good agreement with true labels)")
                elif ari > 0.5:
                    print("(Moderate agreement with true labels)")
                elif ari > 0:
                    print("(Weak agreement with true labels)")
                else:
                    print("(No better than random)")
                
                # Normalized Mutual Information
                nmi = normalized_mutual_info_score(true_labels[valid_mask], labels[valid_mask])
                metrics['normalized_mutual_info'] = nmi
                print(f"\nNormalized Mutual Information: {nmi:.4f}")
                print(f"  Interpretation: {nmi:.4f} ", end="")
                if nmi > 0.7:
                    print("(Strong information shared with true labels)")
                elif nmi > 0.5:
                    print("(Moderate information shared with true labels)")
                elif nmi > 0:
                    print("(Weak information shared with true labels)")
                else:
                    print("(No shared information)")
                
                # Confusion between clusters and true labels
                print("\n" + "-" * 80)
                print("CLUSTER vs TRUE LABEL DISTRIBUTION")
                print("-" * 80)
                
                cross_tab = pd.crosstab(
                    pd.Series(labels[valid_mask], name='Cluster'),
                    pd.Series(true_labels[valid_mask], name='True Label')
                )
                print("\nCross-tabulation:")
                display(cross_tab)
                
                # Percentage breakdown
                print("\nPercentage of each true label in each cluster:")
                cross_tab_pct = cross_tab.div(cross_tab.sum(axis=1), axis=0) * 100
                display(cross_tab_pct.round(1))
                
            except Exception as e:
                print(f"Could not compute external metrics: {e}")
    
    print("\n" + "=" * 80)
    
    return metrics

# Evaluate DBSCAN
if X_for_clustering is not None and 'dbscan_labels' in locals():
    dbscan_metrics = evaluate_clustering(X_for_clustering, dbscan_labels, 
                                        true_labels, method_name="DBSCAN")

================================================================================
EVALUATING DBSCAN
================================================================================

Clusters: 294
Noise points: 9481 (4.2%)

--------------------------------------------------------------------------------
INTERNAL METRICS (no labels used)
--------------------------------------------------------------------------------
Silhouette Score: 0.3546
  Interpretation: 0.3546 (Good - reasonable cluster structure)

Davies-Bouldin Index: 0.5187
  Interpretation: 0.5187 (lower is better)

Calinski-Harabasz Index: 19274.96
  Interpretation: 19274.96 (higher is better)

--------------------------------------------------------------------------------
EXTERNAL METRICS (validation against true labels)
⚠️  These are for VALIDATION only - labels were NOT used in clustering!
--------------------------------------------------------------------------------

Adjusted Rand Index: 0.2070
  Interpretation: 0.2070 (Weak agreement with true labels)

Normalized Mutual Information: 0.3311
  Interpretation: 0.3311 (Weak information shared with true labels)

--------------------------------------------------------------------------------
CLUSTER vs TRUE LABEL DISTRIBUTION
--------------------------------------------------------------------------------

Cross-tabulation:

Percentage of each true label in each cluster:

================================================================================

# Evaluate Hierarchical Clustering
if X_for_clustering is not None and 'hierarchical_labels' in locals():
    # Check if hierarchical clustering was done on sampled data
    if 'sample_indices' in locals() and sample_indices is not None:
        print("\n" + "=" * 80)
        print("NOTE: Hierarchical clustering was performed on sampled data")
        print("=" * 80)
        print(f"Evaluating on {len(sample_indices):,} sampled points\n")
        
        hierarchical_metrics = evaluate_clustering(
            X_for_clustering[sample_indices], 
            hierarchical_labels, 
            true_labels[sample_indices] if true_labels is not None else None,
            method_name="Hierarchical Clustering (Sampled)"
        )
    else:
        hierarchical_metrics = evaluate_clustering(
            X_for_clustering, 
            hierarchical_labels, 
            true_labels, 
            method_name="Hierarchical Clustering"
        )

================================================================================
NOTE: Hierarchical clustering was performed on sampled data
================================================================================
Evaluating on 10,000 sampled points

================================================================================
EVALUATING HIERARCHICAL CLUSTERING (SAMPLED)
================================================================================

Clusters: 5

--------------------------------------------------------------------------------
INTERNAL METRICS (no labels used)
--------------------------------------------------------------------------------
Silhouette Score: 0.4080
  Interpretation: 0.4080 (Good - reasonable cluster structure)

Davies-Bouldin Index: 1.1513
  Interpretation: 1.1513 (lower is better)

Calinski-Harabasz Index: 1684.97
  Interpretation: 1684.97 (higher is better)

--------------------------------------------------------------------------------
EXTERNAL METRICS (validation against true labels)
⚠️  These are for VALIDATION only - labels were NOT used in clustering!
--------------------------------------------------------------------------------

Adjusted Rand Index: 0.1099
  Interpretation: 0.1099 (Weak agreement with true labels)

Normalized Mutual Information: 0.2155
  Interpretation: 0.2155 (Weak information shared with true labels)

--------------------------------------------------------------------------------
CLUSTER vs TRUE LABEL DISTRIBUTION
--------------------------------------------------------------------------------

Cross-tabulation:

Percentage of each true label in each cluster:

================================================================================

def compare_methods(dbscan_metrics, hierarchical_metrics):
    """
    Compare two clustering methods side-by-side.
    """
    print("=" * 80)
    print("METHOD COMPARISON")
    print("=" * 80)
    
    comparison_df = pd.DataFrame({
        'DBSCAN': dbscan_metrics,
        'Hierarchical': hierarchical_metrics
    })
    
    print("\nMetric Comparison:")
    display(comparison_df)
    
    print("\n" + "-" * 80)
    print("INTERPRETATION")
    print("-" * 80)
    
    # Compare internal metrics
    if 'silhouette' in comparison_df.index:
        best_silhouette = comparison_df.loc['silhouette'].idxmax()
        print(f"\nBest Silhouette Score: {best_silhouette} "
              f"({comparison_df.loc['silhouette', best_silhouette]:.4f})")
    
    if 'davies_bouldin' in comparison_df.index:
        best_db = comparison_df.loc['davies_bouldin'].idxmin()
        print(f"Best Davies-Bouldin Index: {best_db} "
              f"({comparison_df.loc['davies_bouldin', best_db]:.4f}) - lower is better")
    
    if 'calinski_harabasz' in comparison_df.index:
        best_ch = comparison_df.loc['calinski_harabasz'].idxmax()
        print(f"Best Calinski-Harabasz Index: {best_ch} "
              f"({comparison_df.loc['calinski_harabasz', best_ch]:.2f})")
    
    # Compare external metrics
    if 'adjusted_rand_index' in comparison_df.index:
        best_ari = comparison_df.loc['adjusted_rand_index'].idxmax()
        print(f"\nBest Agreement with True Labels (ARI): {best_ari} "
              f"({comparison_df.loc['adjusted_rand_index', best_ari]:.4f})")
    
    if 'normalized_mutual_info' in comparison_df.index:
        best_nmi = comparison_df.loc['normalized_mutual_info'].idxmax()
        print(f"Best Information Shared (NMI): {best_nmi} "
              f"({comparison_df.loc['normalized_mutual_info', best_nmi]:.4f})")
    
    print("\n" + "=" * 80)

# Compare methods
if 'dbscan_metrics' in locals() and 'hierarchical_metrics' in locals():
    compare_methods(dbscan_metrics, hierarchical_metrics)

================================================================================
METHOD COMPARISON
================================================================================

Metric Comparison:

--------------------------------------------------------------------------------
INTERPRETATION
--------------------------------------------------------------------------------

Best Silhouette Score: Hierarchical (0.4080)
Best Davies-Bouldin Index: DBSCAN (0.5187) - lower is better
Best Calinski-Harabasz Index: DBSCAN (19274.96)

Best Agreement with True Labels (ARI): DBSCAN (0.2070)
Best Information Shared (NMI): DBSCAN (0.3311)

================================================================================

def analyze_noise_points(dbscan_labels, true_labels, original_df):
    """
    Analyze characteristics of noise points identified by DBSCAN.
    """
    print("=" * 80)
    print("DBSCAN NOISE POINT ANALYSIS")
    print("=" * 80)
    
    noise_mask = dbscan_labels == -1
    n_noise = noise_mask.sum()
    
    if n_noise == 0:
        print("\nNo noise points detected.")
        return
    
    print(f"\nTotal noise points: {n_noise:,} ({n_noise/len(dbscan_labels)*100:.1f}%)")
    
    # What are the true labels of noise points?
    if true_labels is not None:
        print("\n" + "-" * 80)
        print("TRUE LABEL DISTRIBUTION OF NOISE POINTS")
        print("-" * 80)
        
        noise_true_labels = true_labels[noise_mask]
        label_counts = pd.Series(noise_true_labels).value_counts()
        
        print("\nNoise points by true label:")
        for label, count in label_counts.items():
            pct = count / len(noise_true_labels) * 100
            pct_of_total = count / len(true_labels) * 100
            print(f"  {label}: {count:,} ({pct:.1f}% of noise, {pct_of_total:.1f}% of all data)")
        
        # Compare to overall label distribution
        print("\n" + "-" * 80)
        print("COMPARISON TO OVERALL DISTRIBUTION")
        print("-" * 80)
        
        overall_dist = pd.Series(true_labels).value_counts(normalize=True) * 100
        noise_dist = pd.Series(noise_true_labels).value_counts(normalize=True) * 100
        
        comparison = pd.DataFrame({
            'Overall %': overall_dist,
            'Noise %': noise_dist,
            'Difference': noise_dist - overall_dist
        })
        
        print("\nLabel distribution comparison:")
        display(comparison.round(2))
        
        print("\n💡 Interpretation:")
        print("   Positive difference = This label is over-represented in noise")
        print("   Negative difference = This label is under-represented in noise")
    
    print("\n" + "=" * 80)

# Analyze noise points
if 'dbscan_labels' in locals() and df is not None:
    analyze_noise_points(dbscan_labels, true_labels, df)

================================================================================
DBSCAN NOISE POINT ANALYSIS
================================================================================

Total noise points: 9,481 (4.2%)

--------------------------------------------------------------------------------
TRUE LABEL DISTRIBUTION OF NOISE POINTS
--------------------------------------------------------------------------------

Noise points by true label:
  BENIGN: 9,274 (97.8% of noise, 4.1% of all data)
  DDoS: 207 (2.2% of noise, 0.1% of all data)

--------------------------------------------------------------------------------
COMPARISON TO OVERALL DISTRIBUTION
--------------------------------------------------------------------------------

Label distribution comparison:

💡 Interpretation:
   Positive difference = This label is over-represented in noise
   Negative difference = This label is under-represented in noise

================================================================================

print("=" * 80)
print("CLUSTERING ANALYSIS SUMMARY")
print("=" * 80)

print("""
KEY FINDINGS:

1. PREPROCESSING:
   - Started with 80 numerical features
   - Removed highly correlated features (threshold=0.95)
   - Applied PCA for dimensionality reduction
   - Scaled all features for distance-based algorithms

2. DBSCAN RESULTS:
   - Automatically discovered clusters without specifying k
   - Identified noise points that may represent novel attacks
   - Handles irregular cluster shapes and varying densities
   - Outliers are flagged rather than forced into clusters

3. HIERARCHICAL CLUSTERING RESULTS:
   - Revealed hierarchical relationships between attack types
   - Dendrogram shows cluster merge patterns
   - Can explore different granularities by cutting at different heights
   - More interpretable cluster relationships

4. VALIDATION:
   - Both methods show [FILL IN based on your results]
   - Noise points in DBSCAN primarily consist of [FILL IN]
   - Internal metrics suggest [FILL IN]

RECOMMENDATIONS FOR CYBERSECURITY APPLICATION:

1. DBSCAN for Anomaly Detection:
   - Use noise points as potential new attack indicators
   - Investigate high-density clusters for common attack patterns
   - Monitor cluster membership over time for emerging threats

2. Hierarchical Clustering for Attack Taxonomy:
   - Use dendrogram to understand attack type relationships
   - Cut at different heights for different classification granularities
   - Helpful for organizing and categorizing threat intelligence

3. Combined Approach:
   - Use DBSCAN for initial discovery and outlier detection
   - Use Hierarchical Clustering for understanding relationships
   - Cross-reference results to identify robust patterns

NEXT STEPS:

1. Feature Analysis:
   - Examine which features distinguish different clusters
   - Identify key characteristics of each cluster
   - Build interpretable rules from cluster patterns

2. Temporal Analysis:
   - Track cluster evolution over time
   - Identify emerging attack patterns
   - Monitor for drift in attack strategies

3. Supervised Learning:
   - Use cluster insights to improve labeled datasets
   - Train classifiers on well-defined clusters
   - Use clustering as feature engineering step
""")

print("=" * 80)

================================================================================
CLUSTERING ANALYSIS SUMMARY
================================================================================

KEY FINDINGS:

1. PREPROCESSING:
   - Started with 80 numerical features
   - Removed highly correlated features (threshold=0.95)
   - Applied PCA for dimensionality reduction
   - Scaled all features for distance-based algorithms

2. DBSCAN RESULTS:
   - Automatically discovered clusters without specifying k
   - Identified noise points that may represent novel attacks
   - Handles irregular cluster shapes and varying densities
   - Outliers are flagged rather than forced into clusters

3. HIERARCHICAL CLUSTERING RESULTS:
   - Revealed hierarchical relationships between attack types
   - Dendrogram shows cluster merge patterns
   - Can explore different granularities by cutting at different heights
   - More interpretable cluster relationships

4. VALIDATION:
   - Both methods show [FILL IN based on your results]
   - Noise points in DBSCAN primarily consist of [FILL IN]
   - Internal metrics suggest [FILL IN]

RECOMMENDATIONS FOR CYBERSECURITY APPLICATION:

1. DBSCAN for Anomaly Detection:
   - Use noise points as potential new attack indicators
   - Investigate high-density clusters for common attack patterns
   - Monitor cluster membership over time for emerging threats

2. Hierarchical Clustering for Attack Taxonomy:
   - Use dendrogram to understand attack type relationships
   - Cut at different heights for different classification granularities
   - Helpful for organizing and categorizing threat intelligence

3. Combined Approach:
   - Use DBSCAN for initial discovery and outlier detection
   - Use Hierarchical Clustering for understanding relationships
   - Cross-reference results to identify robust patterns

NEXT STEPS:

1. Feature Analysis:
   - Examine which features distinguish different clusters
   - Identify key characteristics of each cluster
   - Build interpretable rules from cluster patterns

2. Temporal Analysis:
   - Track cluster evolution over time
   - Identify emerging attack patterns
   - Monitor for drift in attack strategies

3. Supervised Learning:
   - Use cluster insights to improve labeled datasets
   - Train classifiers on well-defined clusters
   - Use clustering as feature engineering step

================================================================================

def create_binary_labels(df, label_column='Label', benign_value='BENIGN'):
    """
    Convert multi-class labels to binary classification.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset with labels
    label_column : str
        Name of the label column
    benign_value : str
        Value that represents benign/normal traffic
    
    Returns:
    --------
    df_binary : pandas DataFrame
        DataFrame with added binary label column
    """
    print("=" * 80)
    print("CREATING BINARY LABELS")
    print("=" * 80)
    
    # Check if label column exists
    if label_column not in df.columns:
        print(f"\n❌ Error: Column '{label_column}' not found in dataframe")
        print(f"   Available columns: {list(df.columns)}")
        return None
    
    # Create a copy to avoid modifying original
    df_binary = df.copy()
    
    # Show original label distribution
    print(f"\nOriginal labels in '{label_column}' column:")
    original_counts = df_binary[label_column].value_counts()
    for label, count in original_counts.items():
        pct = (count / len(df_binary)) * 100
        print(f"  {label}: {count:,} ({pct:.2f}%)")
    
    # Create binary labels
    # 0 = BENIGN, 1 = ATTACK (everything else)
    df_binary['Binary_Label'] = (df_binary[label_column] != benign_value).astype(int)
    
    print(f"\n" + "-" * 80)
    print("BINARY ENCODING")
    print("-" * 80)
    print(f"Class 0 (BENIGN): '{benign_value}'")
    print(f"Class 1 (ATTACK): Everything else")
    
    # Show mapping examples
    print(f"\nMapping examples:")
    unique_labels = df_binary[label_column].unique()
    for label in unique_labels:
        binary_val = 0 if label == benign_value else 1
        print(f"  '{label}' → {binary_val}")
    
    print("\n✓ Binary labels created in 'Binary_Label' column")
    print("=" * 80)
    
    return df_binary

# Create binary labels
if df is not None:
    df_binary = create_binary_labels(df, label_column='Label', benign_value='BENIGN')
    
    # Display first few rows to verify
    if df_binary is not None:
        print("\nVerification - First 10 rows:")
        display(df_binary[['Label', 'Binary_Label']].head(10))

================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  DDoS: 128,027 (56.71%)
  BENIGN: 97,718 (43.29%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0
  'DDoS' → 1

✓ Binary labels created in 'Binary_Label' column
================================================================================

Verification - First 10 rows:

def analyze_class_balance(df_binary, binary_label_col='Binary_Label'):
    """
    Analyze and visualize class balance for binary classification.
    
    Parameters:
    -----------
    df_binary : pandas DataFrame
        DataFrame with binary labels
    binary_label_col : str
        Name of the binary label column
    
    Returns:
    --------
    balance_stats : dict
        Dictionary with balance statistics
    """
    print("=" * 80)
    print("CLASS BALANCE ANALYSIS")
    print("=" * 80)
    
    # Get class counts
    class_counts = df_binary[binary_label_col].value_counts().sort_index()
    total = len(df_binary)
    
    class_0_count = class_counts.get(0, 0)
    class_1_count = class_counts.get(1, 0)
    
    class_0_pct = (class_0_count / total) * 100
    class_1_pct = (class_1_count / total) * 100
    
    # Calculate imbalance ratio
    majority_class = 0 if class_0_count > class_1_count else 1
    minority_class = 1 - majority_class
    
    majority_count = max(class_0_count, class_1_count)
    minority_count = min(class_0_count, class_1_count)
    
    imbalance_ratio = majority_count / minority_count if minority_count > 0 else float('inf')
    
    # Display results
    print(f"\nTotal samples: {total:,}")
    print("\n" + "-" * 80)
    print("CLASS DISTRIBUTION")
    print("-" * 80)
    
    print(f"\nClass 0 (BENIGN):")
    print(f"  Count: {class_0_count:,}")
    print(f"  Percentage: {class_0_pct:.2f}%")
    
    print(f"\nClass 1 (ATTACK):")
    print(f"  Count: {class_1_count:,}")
    print(f"  Percentage: {class_1_pct:.2f}%")
    
    print("\n" + "-" * 80)
    print("BALANCE ANALYSIS")
    print("-" * 80)
    
    print(f"\nMajority class: {majority_class}")
    print(f"Minority class: {minority_class}")
    print(f"\nImbalance ratio: {imbalance_ratio:.2f}:1")
    print(f"  (Majority class has {imbalance_ratio:.2f}x more samples)")
    
    # Classify the imbalance level
    print("\n" + "-" * 80)
    print("IMBALANCE SEVERITY")
    print("-" * 80)
    
    min_pct = min(class_0_pct, class_1_pct)
    
    if min_pct >= 40:
        severity = "BALANCED"
        color = "🟢"
        recommendation = "No special handling needed. Standard algorithms should work well."
    elif min_pct >= 30:
        severity = "SLIGHT IMBALANCE"
        color = "🟡"
        recommendation = "Consider using class weights. Monitor minority class performance."
    elif min_pct >= 20:
        severity = "MODERATE IMBALANCE"
        color = "🟠"
        recommendation = "Use class weights or SMOTE. Focus on precision/recall metrics. Consider ensemble methods."
    elif min_pct >= 10:
        severity = "SIGNIFICANT IMBALANCE"
        color = "🔴"
        recommendation = "Strongly recommend: class weights, resampling (SMOTE/ADASYN), or ensemble methods. Use stratified splitting. Focus on F1, precision, recall, and ROC-AUC metrics."
    else:
        severity = "SEVERE IMBALANCE"
        color = "🔴🔴"
        recommendation = "CRITICAL: Use specialized techniques like SMOTE, ADASYN, or anomaly detection approaches. Consider treating as anomaly detection problem. Use precision-recall curves over ROC curves."
    
    print(f"\n{color} Severity: {severity}")
    print(f"\nRecommendations:")
    print(f"  {recommendation}")
    
    # Create visualizations
    print("\n" + "=" * 80)
    print("VISUALIZATIONS")
    print("=" * 80)
    
    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
    
    # 1. Bar chart
    axes[0, 0].bar(['Class 0\n(BENIGN)', 'Class 1\n(ATTACK)'], 
                   [class_0_count, class_1_count],
                   color=['#2ecc71', '#e74c3c'])
    axes[0, 0].set_ylabel('Number of Samples')
    axes[0, 0].set_title('Class Distribution (Counts)', fontweight='bold')
    axes[0, 0].grid(True, alpha=0.3, axis='y')
    
    # Add count labels on bars
    for i, (label, count) in enumerate(zip(['Class 0', 'Class 1'], [class_0_count, class_1_count])):
        axes[0, 0].text(i, count, f'{count:,}\n({count/total*100:.1f}%)', 
                       ha='center', va='bottom', fontweight='bold')
    
    # 2. Pie chart
    colors = ['#2ecc71', '#e74c3c']
    explode = (0.05, 0.05)
    axes[0, 1].pie([class_0_count, class_1_count], 
                   labels=['Class 0\n(BENIGN)', 'Class 1\n(ATTACK)'],
                   autopct='%1.1f%%',
                   colors=colors,
                   explode=explode,
                   shadow=True,
                   startangle=90)
    axes[0, 1].set_title('Class Distribution (Percentage)', fontweight='bold')
    
    # 3. Imbalance ratio visualization
    axes[1, 0].barh(['Minority\nClass', 'Majority\nClass'], 
                    [1, imbalance_ratio],
                    color=['#e74c3c', '#3498db'])
    axes[1, 0].set_xlabel('Relative Size')
    axes[1, 0].set_title(f'Imbalance Ratio: {imbalance_ratio:.2f}:1', fontweight='bold')
    axes[1, 0].grid(True, alpha=0.3, axis='x')
    
    # Add ratio labels
    axes[1, 0].text(1, 0, '1.0x (baseline)', va='center', ha='left', fontweight='bold')
    axes[1, 0].text(imbalance_ratio, 1, f'{imbalance_ratio:.2f}x', va='center', ha='left', fontweight='bold')
    
    # 4. Severity indicator
    axes[1, 1].axis('off')
    
    # Create severity gauge
    severity_text = f"{color} {severity}\n\n"
    severity_text += f"Minority class: {min_pct:.1f}%\n"
    severity_text += f"Imbalance ratio: {imbalance_ratio:.2f}:1\n\n"
    severity_text += "Recommendation:\n" + "\n".join(["  " + line for line in recommendation.split(". ")])
    
    axes[1, 1].text(0.5, 0.5, severity_text, 
                   ha='center', va='center',
                   fontsize=10,
                   bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))
    axes[1, 1].set_title('Balance Assessment', fontweight='bold')
    
    plt.tight_layout()
    plt.show()
    
    # Prepare statistics dictionary
    balance_stats = {
        'total_samples': total,
        'class_0_count': class_0_count,
        'class_1_count': class_1_count,
        'class_0_percentage': class_0_pct,
        'class_1_percentage': class_1_pct,
        'imbalance_ratio': imbalance_ratio,
        'majority_class': majority_class,
        'minority_class': minority_class,
        'severity': severity,
        'recommendation': recommendation
    }
    
    print("\n" + "=" * 80)
    
    return balance_stats

# Analyze class balance
if 'df_binary' in locals() and df_binary is not None:
    balance_stats = analyze_class_balance(df_binary, binary_label_col='Binary_Label')

================================================================================
CLASS BALANCE ANALYSIS
================================================================================

Total samples: 225,745

--------------------------------------------------------------------------------
CLASS DISTRIBUTION
--------------------------------------------------------------------------------

Class 0 (BENIGN):
  Count: 97,718
  Percentage: 43.29%

Class 1 (ATTACK):
  Count: 128,027
  Percentage: 56.71%

--------------------------------------------------------------------------------
BALANCE ANALYSIS
--------------------------------------------------------------------------------

Majority class: 1
Minority class: 0

Imbalance ratio: 1.31:1
  (Majority class has 1.31x more samples)

--------------------------------------------------------------------------------
IMBALANCE SEVERITY
--------------------------------------------------------------------------------

🟢 Severity: BALANCED

Recommendations:
  No special handling needed. Standard algorithms should work well.

================================================================================
VISUALIZATIONS
================================================================================

================================================================================

def print_ml_implications(balance_stats):
    """
    Print ML implications based on class balance.
    """
    print("=" * 80)
    print("MACHINE LEARNING IMPLICATIONS")
    print("=" * 80)
    
    severity = balance_stats['severity']
    imbalance_ratio = balance_stats['imbalance_ratio']
    minority_class = balance_stats['minority_class']
    
    print("\n📊 Dataset Characteristics:")
    print(f"  Total samples: {balance_stats['total_samples']:,}")
    print(f"  Imbalance ratio: {imbalance_ratio:.2f}:1")
    print(f"  Minority class ({minority_class}): {balance_stats[f'class_{minority_class}_percentage']:.2f}%")
    print(f"  Severity: {severity}")
    
    print("\n" + "-" * 80)
    print("1. TRAIN-TEST SPLIT")
    print("-" * 80)
    print("\n✓ MUST use stratified splitting:")
    print("  from sklearn.model_selection import train_test_split")
    print("  X_train, X_test, y_train, y_test = train_test_split(")
    print("      X, y, test_size=0.2, stratify=y, random_state=42")
    print("  )")
    print("\n  This ensures both train and test sets have same class proportions.")
    
    print("\n" + "-" * 80)
    print("2. EVALUATION METRICS")
    print("-" * 80)
    print("\n❌ DON'T rely solely on accuracy!")
    print(f"  With {imbalance_ratio:.1f}:1 imbalance, a model that always predicts")
    print(f"  the majority class achieves {max(balance_stats['class_0_percentage'], balance_stats['class_1_percentage']):.1f}% accuracy.")
    
    print("\n✓ DO use these metrics:")
    print("  • Confusion Matrix - see all types of errors")
    print("  • Precision - of predicted attacks, how many are real?")
    print("  • Recall - of real attacks, how many did we catch?")
    print("  • F1-Score - harmonic mean of precision and recall")
    print("  • ROC-AUC - overall discriminative ability")
    print("  • Precision-Recall Curve - especially for severe imbalance")
    
    print("\n" + "-" * 80)
    print("3. HANDLING IMBALANCE")
    print("-" * 80)
    
    if severity in ['BALANCED', 'SLIGHT IMBALANCE']:
        print("\n✓ Your dataset is relatively balanced.")
        print("  Standard algorithms should work well with minimal adjustments.")
        print("\n  Optional: Use class weights for slight boost:")
        print("    model = RandomForestClassifier(class_weight='balanced')")
    
    elif severity == 'MODERATE IMBALANCE':
        print("\n⚠️ Moderate imbalance detected. Recommended approaches:")
        print("\n  A. Class Weights (Easiest):")
        print("     model = RandomForestClassifier(class_weight='balanced')")
        print("     model = SVC(class_weight='balanced')")
        print("\n  B. SMOTE Resampling:")
        print("     from imblearn.over_sampling import SMOTE")
        print("     smote = SMOTE(random_state=42)")
        print("     X_resampled, y_resampled = smote.fit_resample(X_train, y_train)")
    
    else:  # SIGNIFICANT or SEVERE
        print("\n🔴 Significant imbalance! Multiple techniques recommended:")
        print("\n  A. SMOTE or ADASYN (Synthetic oversampling):")
        print("     from imblearn.over_sampling import SMOTE, ADASYN")
        print("     smote = SMOTE(random_state=42)")
        print("     X_resampled, y_resampled = smote.fit_resample(X_train, y_train)")
        print("\n  B. Class Weights:")
        print("     model = RandomForestClassifier(class_weight='balanced')")
        print("\n  C. Ensemble Methods:")
        print("     from imblearn.ensemble import BalancedRandomForestClassifier")
        print("     model = BalancedRandomForestClassifier()")
        print("\n  D. Consider Anomaly Detection:")
        print("     from sklearn.ensemble import IsolationForest")
        print("     model = IsolationForest()")
    
    print("\n" + "-" * 80)
    print("4. ALGORITHM SELECTION")
    print("-" * 80)
    print("\n✓ Good choices for imbalanced data:")
    print("  • Random Forest (with class_weight='balanced')")
    print("  • XGBoost (with scale_pos_weight parameter)")
    print("  • Ensemble methods from imbalanced-learn library")
    
    print("\n⚠️ Be careful with:")
    print("  • Logistic Regression - needs class weights")
    print("  • KNN - sensitive to imbalance")
    print("  • Naive Bayes - can work but monitor carefully")
    
    print("\n" + "-" * 80)
    print("5. CYBERSECURITY CONTEXT")
    print("-" * 80)
    print("\nIn cybersecurity (intrusion detection):")
    print("  • False Negatives (missed attacks) are usually MORE costly")
    print("  • False Positives (false alarms) cause alert fatigue")
    print("  • Balance depends on context: high-security vs general monitoring")
    print("\n  Recommendation: Optimize for high RECALL (catch attacks)")
    print("                  while maintaining acceptable PRECISION (limit false alarms)")
    
    print("\n" + "=" * 80)
    print("\n💡 NEXT STEPS:")
    print("  1. Split data with stratification")
    print("  2. Choose and train baseline model")
    print("  3. Evaluate with appropriate metrics")
    print("  4. Apply imbalance handling if needed")
    print("  5. Compare multiple models")
    print("  6. Tune hyperparameters")
    print("  7. Final evaluation on test set")
    print("=" * 80)

# Print ML implications
if 'balance_stats' in locals():
    print_ml_implications(balance_stats)

================================================================================
MACHINE LEARNING IMPLICATIONS
================================================================================

📊 Dataset Characteristics:
  Total samples: 225,745
  Imbalance ratio: 1.31:1
  Minority class (0): 43.29%
  Severity: BALANCED

--------------------------------------------------------------------------------
1. TRAIN-TEST SPLIT
--------------------------------------------------------------------------------

✓ MUST use stratified splitting:
  from sklearn.model_selection import train_test_split
  X_train, X_test, y_train, y_test = train_test_split(
      X, y, test_size=0.2, stratify=y, random_state=42
  )

  This ensures both train and test sets have same class proportions.

--------------------------------------------------------------------------------
2. EVALUATION METRICS
--------------------------------------------------------------------------------

❌ DON'T rely solely on accuracy!
  With 1.3:1 imbalance, a model that always predicts
  the majority class achieves 56.7% accuracy.

✓ DO use these metrics:
  • Confusion Matrix - see all types of errors
  • Precision - of predicted attacks, how many are real?
  • Recall - of real attacks, how many did we catch?
  • F1-Score - harmonic mean of precision and recall
  • ROC-AUC - overall discriminative ability
  • Precision-Recall Curve - especially for severe imbalance

--------------------------------------------------------------------------------
3. HANDLING IMBALANCE
--------------------------------------------------------------------------------

✓ Your dataset is relatively balanced.
  Standard algorithms should work well with minimal adjustments.

  Optional: Use class weights for slight boost:
    model = RandomForestClassifier(class_weight='balanced')

--------------------------------------------------------------------------------
4. ALGORITHM SELECTION
--------------------------------------------------------------------------------

✓ Good choices for imbalanced data:
  • Random Forest (with class_weight='balanced')
  • XGBoost (with scale_pos_weight parameter)
  • Ensemble methods from imbalanced-learn library

⚠️ Be careful with:
  • Logistic Regression - needs class weights
  • KNN - sensitive to imbalance
  • Naive Bayes - can work but monitor carefully

--------------------------------------------------------------------------------
5. CYBERSECURITY CONTEXT
--------------------------------------------------------------------------------

In cybersecurity (intrusion detection):
  • False Negatives (missed attacks) are usually MORE costly
  • False Positives (false alarms) cause alert fatigue
  • Balance depends on context: high-security vs general monitoring

  Recommendation: Optimize for high RECALL (catch attacks)
                  while maintaining acceptable PRECISION (limit false alarms)

================================================================================

💡 NEXT STEPS:
  1. Split data with stratification
  2. Choose and train baseline model
  3. Evaluate with appropriate metrics
  4. Apply imbalance handling if needed
  5. Compare multiple models
  6. Tune hyperparameters
  7. Final evaluation on test set
================================================================================

# Uncomment to save the dataset with binary labels
# if 'df_binary' in locals() and df_binary is not None:
#     output_file = 'network_traffic_binary.csv'
#     df_binary.to_csv(output_file, index=False)
#     print(f"✓ Saved dataset with binary labels to: {output_file}")
#     print(f"  Shape: {df_binary.shape}")
#     print(f"  Columns include: {list(df_binary.columns[-5:])}")

# Define important port numbers based on IANA assignments and common services
IMPORTANT_PORTS = {
    # Web Services
    80: 'HTTP',
    443: 'HTTPS',
    8080: 'HTTP_Proxy',
    8443: 'HTTPS_Alt',
    
    # Remote Access
    22: 'SSH',
    23: 'Telnet',
    3389: 'RDP',
    5900: 'VNC',
    
    # DNS
    53: 'DNS',
    
    # Email
    25: 'SMTP',
    110: 'POP3',
    143: 'IMAP',
    587: 'SMTP_Submission',
    993: 'IMAPS',
    995: 'POP3S',
    
    # File Transfer
    20: 'FTP_Data',
    21: 'FTP_Control',
    69: 'TFTP',
    445: 'SMB',
    
    # Database
    3306: 'MySQL',
    5432: 'PostgreSQL',
    1521: 'Oracle',
    1433: 'MSSQL',
    27017: 'MongoDB',
    6379: 'Redis',
    
    # Network Services
    67: 'DHCP_Server',
    68: 'DHCP_Client',
    123: 'NTP',
    161: 'SNMP',
    162: 'SNMP_Trap',
    
    # Other Common
    137: 'NetBIOS_Name',
    138: 'NetBIOS_Datagram',
    139: 'NetBIOS_Session',
    389: 'LDAP',
    636: 'LDAPS',
    514: 'Syslog',
    
    # Gaming/Streaming (often DDoS targets)
    27015: 'Steam',
    3074: 'Xbox_Live',
    5060: 'SIP',
    
    # Cloud/Container
    2375: 'Docker',
    2376: 'Docker_TLS',
    6443: 'Kubernetes_API',
}

print("=" * 80)
print("IMPORTANT PORT DEFINITIONS")
print("=" * 80)
print(f"\nDefined {len(IMPORTANT_PORTS)} important ports for analysis:")
print("\nBy category:")

# Group by category for display
categories = {
    'Web Services': [80, 443, 8080, 8443],
    'Remote Access': [22, 23, 3389, 5900],
    'DNS': [53],
    'Email': [25, 110, 143, 587, 993, 995],
    'File Transfer': [20, 21, 69, 445],
    'Database': [3306, 5432, 1521, 1433, 27017, 6379],
    'Network Services': [67, 68, 123, 161, 162, 137, 138, 139, 389, 636, 514]
}

for category, ports in categories.items():
    print(f"\n{category}:")
    for port in ports:
        if port in IMPORTANT_PORTS:
            print(f"  Port {port:5d}: {IMPORTANT_PORTS[port]}")

print("\n" + "=" * 80)

================================================================================
IMPORTANT PORT DEFINITIONS
================================================================================

Defined 42 important ports for analysis:

By category:

Web Services:
  Port    80: HTTP
  Port   443: HTTPS
  Port  8080: HTTP_Proxy
  Port  8443: HTTPS_Alt

Remote Access:
  Port    22: SSH
  Port    23: Telnet
  Port  3389: RDP
  Port  5900: VNC

DNS:
  Port    53: DNS

Email:
  Port    25: SMTP
  Port   110: POP3
  Port   143: IMAP
  Port   587: SMTP_Submission
  Port   993: IMAPS
  Port   995: POP3S

File Transfer:
  Port    20: FTP_Data
  Port    21: FTP_Control
  Port    69: TFTP
  Port   445: SMB

Database:
  Port  3306: MySQL
  Port  5432: PostgreSQL
  Port  1521: Oracle
  Port  1433: MSSQL
  Port 27017: MongoDB
  Port  6379: Redis

Network Services:
  Port    67: DHCP_Server
  Port    68: DHCP_Client
  Port   123: NTP
  Port   161: SNMP
  Port   162: SNMP_Trap
  Port   137: NetBIOS_Name
  Port   138: NetBIOS_Datagram
  Port   139: NetBIOS_Session
  Port   389: LDAP
  Port   636: LDAPS
  Port   514: Syslog

================================================================================

def analyze_port_distribution(df, port_column, top_n=20):
    """
    Analyze port number distribution in the dataset.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset
    port_column : str
        Name of port column to analyze
    top_n : int
        Number of top ports to display
    
    Returns:
    --------
    port_stats : dict
        Statistics about ports
    """
    print("=" * 80)
    print(f"PORT DISTRIBUTION ANALYSIS: {port_column}")
    print("=" * 80)
    
    # Get port value counts
    port_counts = df[port_column].value_counts()
    
    print(f"\nTotal unique ports: {len(port_counts):,}")
    print(f"Total samples: {len(df):,}")
    
    # Statistics
    print(f"\nPort range: {df[port_column].min():.0f} to {df[port_column].max():.0f}")
    print(f"Mean port: {df[port_column].mean():.2f}")
    print(f"Median port: {df[port_column].median():.0f}")
    
    # Top ports
    print(f"\n" + "-" * 80)
    print(f"TOP {top_n} MOST COMMON PORTS")
    print("-" * 80)
    print(f"\n{'Port':<8} {'Count':<12} {'Percentage':<12} {'Service Name'}")
    print("-" * 80)
    
    for port, count in port_counts.head(top_n).items():
        pct = (count / len(df)) * 100
        service = IMPORTANT_PORTS.get(int(port), 'Unknown')
        print(f"{int(port):<8} {count:<12,} {pct:<11.2f}% {service}")
    
    # Check which important ports are in the data
    print("\n" + "-" * 80)
    print("IMPORTANT PORTS PRESENT IN DATA")
    print("-" * 80)
    
    present_important = []
    absent_important = []
    
    for port, name in IMPORTANT_PORTS.items():
        if port in port_counts.index:
            count = port_counts[port]
            pct = (count / len(df)) * 100
            present_important.append((port, name, count, pct))
        else:
            absent_important.append((port, name))
    
    print(f"\nFound {len(present_important)} out of {len(IMPORTANT_PORTS)} important ports:")
    print(f"\n{'Port':<8} {'Count':<12} {'Percentage':<12} {'Service Name'}")
    print("-" * 80)
    
    for port, name, count, pct in sorted(present_important, key=lambda x: x[2], reverse=True)[:20]:
        print(f"{port:<8} {count:<12,} {pct:<11.2f}% {name}")
    
    if len(present_important) > 20:
        print(f"\n... and {len(present_important) - 20} more")
    
    # Visualize
    print("\n" + "=" * 80)
    print("VISUALIZATION")
    print("=" * 80)
    
    fig, axes = plt.subplots(2, 2, figsize=(16, 12))
    
    # 1. Top ports bar chart
    top_ports = port_counts.head(20)
    axes[0, 0].barh(range(len(top_ports)), top_ports.values)
    axes[0, 0].set_yticks(range(len(top_ports)))
    axes[0, 0].set_yticklabels([f"Port {int(p)} ({IMPORTANT_PORTS.get(int(p), 'Unknown')})" 
                                 for p in top_ports.index])
    axes[0, 0].set_xlabel('Number of Occurrences')
    axes[0, 0].set_title(f'Top 20 Ports in {port_column}', fontweight='bold')
    axes[0, 0].invert_yaxis()
    axes[0, 0].grid(True, alpha=0.3, axis='x')
    
    # 2. Port range distribution
    axes[0, 1].hist(df[port_column], bins=100, edgecolor='black', alpha=0.7)
    axes[0, 1].set_xlabel('Port Number')
    axes[0, 1].set_ylabel('Frequency')
    axes[0, 1].set_title(f'Port Distribution - {port_column}', fontweight='bold')
    axes[0, 1].grid(True, alpha=0.3)
    
    # Add well-known port boundary
    axes[0, 1].axvline(1024, color='r', linestyle='--', linewidth=2, label='Well-known ports boundary')
    axes[0, 1].legend()
    
    # 3. Important ports present
    if len(present_important) > 0:
        # Get top 15 by count
        top_important = sorted(present_important, key=lambda x: x[2], reverse=True)[:15]
        ports_list = [f"Port {p}\n{n}" for p, n, c, pct in top_important]
        counts_list = [c for p, n, c, pct in top_important]
        
        axes[1, 0].barh(range(len(top_important)), counts_list)
        axes[1, 0].set_yticks(range(len(top_important)))
        axes[1, 0].set_yticklabels(ports_list, fontsize=8)
        axes[1, 0].set_xlabel('Number of Occurrences')
        axes[1, 0].set_title(f'Top Important Ports in {port_column}', fontweight='bold')
        axes[1, 0].invert_yaxis()
        axes[1, 0].grid(True, alpha=0.3, axis='x')
    else:
        axes[1, 0].text(0.5, 0.5, 'No important ports found', 
                       ha='center', va='center', transform=axes[1, 0].transAxes)
        axes[1, 0].set_title(f'Important Ports in {port_column}', fontweight='bold')
    
    # 4. Coverage statistics
    well_known_mask = df[port_column] < 1024
    registered_mask = (df[port_column] >= 1024) & (df[port_column] < 49152)
    dynamic_mask = df[port_column] >= 49152
    
    port_ranges = {
        'Well-Known\n(0-1023)': well_known_mask.sum(),
        'Registered\n(1024-49151)': registered_mask.sum(),
        'Dynamic\n(49152-65535)': dynamic_mask.sum()
    }
    
    colors = ['#e74c3c', '#3498db', '#2ecc71']
    axes[1, 1].pie(port_ranges.values(), labels=port_ranges.keys(), autopct='%1.1f%%',
                   colors=colors, startangle=90, shadow=True)
    axes[1, 1].set_title(f'Port Range Distribution - {port_column}', fontweight='bold')
    
    plt.tight_layout()
    plt.show()
    
    # Return statistics
    port_stats = {
        'column': port_column,
        'unique_ports': len(port_counts),
        'total_samples': len(df),
        'top_ports': port_counts.head(top_n).to_dict(),
        'important_ports_present': len(present_important),
        'important_ports_total': len(IMPORTANT_PORTS),
        'present_important_ports': present_important,
        'well_known_count': well_known_mask.sum(),
        'registered_count': registered_mask.sum(),
        'dynamic_count': dynamic_mask.sum()
    }
    
    print("\n" + "=" * 80)
    
    return port_stats

# Analyze source and destination ports
if 'df_binary' in locals() and df_binary is not None:
    print("\n" + "#" * 80)
    print("# ANALYZING SOURCE PORT")
    print("#" * 80 + "\n")
    src_port_stats = analyze_port_distribution(df_binary, 'Source Port', top_n=20)
    
    print("\n\n" + "#" * 80)
    print("# ANALYZING DESTINATION PORT")
    print("#" * 80 + "\n")
    dst_port_stats = analyze_port_distribution(df_binary, 'Destination Port', top_n=20)

################################################################################
# ANALYZING SOURCE PORT
################################################################################

================================================================================
PORT DISTRIBUTION ANALYSIS: Source Port
================================================================================

Total unique ports: 50,697
Total samples: 225,745

Port range: 0 to 65534
Mean port: 38257.57
Median port: 49799

--------------------------------------------------------------------------------
TOP 20 MOST COMMON PORTS
--------------------------------------------------------------------------------

Port     Count        Percentage   Service Name
--------------------------------------------------------------------------------
80       33,528       14.85      % HTTP
443      6,419        2.84       % HTTPS
123      362          0.16       % NTP
137      274          0.12       % NetBIOS_Name
22       156          0.07       % SSH
21       86           0.04       % FTP_Control
0        54           0.02       % Unknown
138      53           0.02       % NetBIOS_Datagram
139      31           0.01       % NetBIOS_Session
389      26           0.01       % LDAP
39524    22           0.01       % Unknown
60406    20           0.01       % Unknown
48318    20           0.01       % Unknown
60883    20           0.01       % Unknown
61399    19           0.01       % Unknown
56540    18           0.01       % Unknown
49666    18           0.01       % Unknown
60340    17           0.01       % Unknown
62146    17           0.01       % Unknown
61407    17           0.01       % Unknown

--------------------------------------------------------------------------------
IMPORTANT PORTS PRESENT IN DATA
--------------------------------------------------------------------------------

Found 16 out of 42 important ports:

Port     Count        Percentage   Service Name
--------------------------------------------------------------------------------
80       33,528       14.85      % HTTP
443      6,419        2.84       % HTTPS
123      362          0.16       % NTP
137      274          0.12       % NetBIOS_Name
22       156          0.07       % SSH
21       86           0.04       % FTP_Control
138      53           0.02       % NetBIOS_Datagram
139      31           0.01       % NetBIOS_Session
389      26           0.01       % LDAP
445      14           0.01       % SMB
5432     2            0.00       % PostgreSQL
5900     1            0.00       % VNC
53       1            0.00       % DNS
27017    1            0.00       % MongoDB
27015    1            0.00       % Steam
2375     1            0.00       % Docker

================================================================================
VISUALIZATION
================================================================================

================================================================================


################################################################################
# ANALYZING DESTINATION PORT
################################################################################

================================================================================
PORT DISTRIBUTION ANALYSIS: Destination Port
================================================================================

Total unique ports: 23,950
Total samples: 225,745

Port range: 0 to 65532
Mean port: 8879.62
Median port: 80

--------------------------------------------------------------------------------
TOP 20 MOST COMMON PORTS
--------------------------------------------------------------------------------

Port     Count        Percentage   Service Name
--------------------------------------------------------------------------------
80       136,951      60.67      % HTTP
53       31,950       14.15      % DNS
443      13,485       5.97       % HTTPS
8080     510          0.23       % HTTP_Proxy
123      362          0.16       % NTP
22       342          0.15       % SSH
137      274          0.12       % NetBIOS_Name
389      261          0.12       % LDAP
88       173          0.08       % Unknown
21       167          0.07       % FTP_Control
465      147          0.07       % Unknown
139      100          0.04       % NetBIOS_Session
3268     91           0.04       % Unknown
0        54           0.02       % Unknown
138      53           0.02       % NetBIOS_Datagram
445      48           0.02       % SMB
135      23           0.01       % Unknown
49666    23           0.01       % Unknown
5353     16           0.01       % Unknown
49671    12           0.01       % Unknown

--------------------------------------------------------------------------------
IMPORTANT PORTS PRESENT IN DATA
--------------------------------------------------------------------------------

Found 15 out of 42 important ports:

Port     Count        Percentage   Service Name
--------------------------------------------------------------------------------
80       136,951      60.67      % HTTP
53       31,950       14.15      % DNS
443      13,485       5.97       % HTTPS
8080     510          0.23       % HTTP_Proxy
123      362          0.16       % NTP
22       342          0.15       % SSH
137      274          0.12       % NetBIOS_Name
389      261          0.12       % LDAP
21       167          0.07       % FTP_Control
139      100          0.04       % NetBIOS_Session
138      53           0.02       % NetBIOS_Datagram
445      48           0.02       % SMB
8443     2            0.00       % HTTPS_Alt
27017    1            0.00       % MongoDB
27015    1            0.00       % Steam

================================================================================
VISUALIZATION
================================================================================

================================================================================

def create_port_features(df, port_column, port_stats, min_occurrences=100):
    """
    Create one-hot encoded features for important ports.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset
    port_column : str
        Name of port column (e.g., 'Source Port' or 'Destination Port')
    port_stats : dict
        Port statistics from analyze_port_distribution
    min_occurrences : int
        Minimum occurrences for a port to get its own feature
    
    Returns:
    --------
    df_with_features : pandas DataFrame
        DataFrame with new port features
    feature_names : list
        Names of created features
    """
    print("=" * 80)
    print(f"CREATING PORT FEATURES FOR: {port_column}")
    print("=" * 80)
    
    df_new = df.copy()
    feature_names = []
    
    # Determine prefix based on column
    if 'Source' in port_column:
        prefix = 'src_port'
    elif 'Destination' in port_column:
        prefix = 'dst_port'
    else:
        prefix = 'port'
    
    # Get important ports that actually appear in data
    present_important = port_stats['present_important_ports']
    
    # Filter by minimum occurrences
    ports_to_encode = [(port, name, count, pct) for port, name, count, pct in present_important 
                       if count >= min_occurrences]
    
    print(f"\nCreating features for {len(ports_to_encode)} ports (min {min_occurrences:,} occurrences)")
    print("\n" + "-" * 80)
    print(f"{'Port':<8} {'Service':<20} {'Count':<12} {'Percentage'}")
    print("-" * 80)
    
    encoded_ports_set = set()
    
    for port, name, count, pct in ports_to_encode:
        # Create binary feature
        feature_name = f"{prefix}_{port}_{name}"
        df_new[feature_name] = (df_new[port_column] == port).astype(int)
        feature_names.append(feature_name)
        encoded_ports_set.add(port)
        
        print(f"{port:<8} {name:<20} {count:<12,} {pct:.2f}%")
    
    # Create 'other_ports' feature for everything else
    other_feature_name = f"{prefix}_other"
    df_new[other_feature_name] = (~df_new[port_column].isin(encoded_ports_set)).astype(int)
    feature_names.append(other_feature_name)
    
    other_count = df_new[other_feature_name].sum()
    other_pct = (other_count / len(df_new)) * 100
    
    print("-" * 80)
    print(f"{'Other':<8} {'(all remaining)':<20} {other_count:<12,} {other_pct:.2f}%")
    
    print(f"\n✓ Created {len(feature_names)} new features")
    print(f"  Feature names: {feature_names[:5]}..." if len(feature_names) > 5 else f"  Feature names: {feature_names}")
    
    # Verify - each row should have exactly one port feature = 1
    port_feature_sum = df_new[feature_names].sum(axis=1)
    if (port_feature_sum == 1).all():
        print(f"\n✓ Verification passed: Each row has exactly one port feature active")
    else:
        print(f"\n⚠️ Warning: Some rows have {port_feature_sum.value_counts().to_dict()} port features active")
    
    print("\n" + "=" * 80)
    
    return df_new, feature_names

# Create port features
if 'df_binary' in locals() and 'src_port_stats' in locals() and 'dst_port_stats' in locals():
    # Source port features
    print("\n" + "#" * 80)
    print("# CREATING SOURCE PORT FEATURES")
    print("#" * 80 + "\n")
    df_with_ports, src_port_features = create_port_features(
        df_binary, 
        'Source Port', 
        src_port_stats,
        min_occurrences=100  # Adjust this threshold as needed
    )
    
    # Destination port features
    print("\n" + "#" * 80)
    print("# CREATING DESTINATION PORT FEATURES")
    print("#" * 80 + "\n")
    df_with_ports, dst_port_features = create_port_features(
        df_with_ports, 
        'Destination Port', 
        dst_port_stats,
        min_occurrences=100
    )
    
    all_port_features = src_port_features + dst_port_features
    
    print("\n" + "=" * 80)
    print("SUMMARY")
    print("=" * 80)
    print(f"\nTotal new port features created: {len(all_port_features)}")
    print(f"  Source port features: {len(src_port_features)}")
    print(f"  Destination port features: {len(dst_port_features)}")
    print(f"\nDataset shape: {df_with_ports.shape}")
    print(f"  Original: {df_binary.shape}")
    print(f"  New features added: {df_with_ports.shape[1] - df_binary.shape[1]}")
    print("=" * 80)

################################################################################
# CREATING SOURCE PORT FEATURES
################################################################################

================================================================================
CREATING PORT FEATURES FOR: Source Port
================================================================================

Creating features for 5 ports (min 100 occurrences)

--------------------------------------------------------------------------------
Port     Service              Count        Percentage
--------------------------------------------------------------------------------
80       HTTP                 33,528       14.85%
443      HTTPS                6,419        2.84%
22       SSH                  156          0.07%
123      NTP                  362          0.16%
137      NetBIOS_Name         274          0.12%
--------------------------------------------------------------------------------
Other    (all remaining)      185,006      81.95%

✓ Created 6 new features
  Feature names: ['src_port_80_HTTP', 'src_port_443_HTTPS', 'src_port_22_SSH', 'src_port_123_NTP', 'src_port_137_NetBIOS_Name']...

✓ Verification passed: Each row has exactly one port feature active

================================================================================

################################################################################
# CREATING DESTINATION PORT FEATURES
################################################################################

================================================================================
CREATING PORT FEATURES FOR: Destination Port
================================================================================

Creating features for 10 ports (min 100 occurrences)

--------------------------------------------------------------------------------
Port     Service              Count        Percentage
--------------------------------------------------------------------------------
80       HTTP                 136,951      60.67%
443      HTTPS                13,485       5.97%
8080     HTTP_Proxy           510          0.23%
22       SSH                  342          0.15%
53       DNS                  31,950       14.15%
21       FTP_Control          167          0.07%
123      NTP                  362          0.16%
137      NetBIOS_Name         274          0.12%
139      NetBIOS_Session      100          0.04%
389      LDAP                 261          0.12%
--------------------------------------------------------------------------------
Other    (all remaining)      41,343       18.31%

✓ Created 11 new features
  Feature names: ['dst_port_80_HTTP', 'dst_port_443_HTTPS', 'dst_port_8080_HTTP_Proxy', 'dst_port_22_SSH', 'dst_port_53_DNS']...

✓ Verification passed: Each row has exactly one port feature active

================================================================================

================================================================================
SUMMARY
================================================================================

Total new port features created: 17
  Source port features: 6
  Destination port features: 11

Dataset shape: (225745, 103)
  Original: (225745, 86)
  New features added: 17
================================================================================

def compare_ports_by_class(df, port_column, class_column='Binary_Label', top_n=15):
    """
    Compare port usage between classes.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset with port and class columns
    port_column : str
        Name of port column
    class_column : str
        Name of binary class column
    top_n : int
        Number of top ports to analyze
    """
    print("=" * 80)
    print(f"PORT COMPARISON BY CLASS: {port_column}")
    print("=" * 80)
    
    # Split by class
    benign = df[df[class_column] == 0]
    attack = df[df[class_column] == 1]
    
    print(f"\nClass 0 (BENIGN): {len(benign):,} samples")
    print(f"Class 1 (ATTACK): {len(attack):,} samples")
    
    # Top ports for each class
    benign_ports = benign[port_column].value_counts().head(top_n)
    attack_ports = attack[port_column].value_counts().head(top_n)
    
    print("\n" + "-" * 80)
    print("TOP PORTS BY CLASS")
    print("-" * 80)
    
    # Create comparison DataFrame
    comparison_data = []
    
    all_top_ports = set(benign_ports.index).union(set(attack_ports.index))
    
    for port in all_top_ports:
        benign_count = benign_ports.get(port, 0)
        attack_count = attack_ports.get(port, 0)
        
        benign_pct = (benign_count / len(benign)) * 100 if len(benign) > 0 else 0
        attack_pct = (attack_count / len(attack)) * 100 if len(attack) > 0 else 0
        
        total_count = benign_count + attack_count
        
        comparison_data.append({
            'Port': int(port),
            'Service': IMPORTANT_PORTS.get(int(port), 'Unknown'),
            'Benign_Count': benign_count,
            'Benign_Pct': benign_pct,
            'Attack_Count': attack_count,
            'Attack_Pct': attack_pct,
            'Total': total_count,
            'Difference': attack_pct - benign_pct
        })
    
    comparison_df = pd.DataFrame(comparison_data).sort_values('Total', ascending=False)
    
    print("\nTop ports (sorted by total occurrences):")
    display(comparison_df.head(20))
    
    # Identify distinctive ports
    print("\n" + "-" * 80)
    print("PORTS MOST ASSOCIATED WITH ATTACKS")
    print("-" * 80)
    
    attack_distinctive = comparison_df[comparison_df['Difference'] > 5].sort_values('Difference', ascending=False)
    
    if len(attack_distinctive) > 0:
        print("\nPorts used significantly more in attacks:")
        print(f"\n{'Port':<8} {'Service':<20} {'Attack %':<12} {'Benign %':<12} {'Difference'}")
        print("-" * 80)
        for _, row in attack_distinctive.head(10).iterrows():
            print(f"{row['Port']:<8} {row['Service']:<20} {row['Attack_Pct']:<11.2f}% "
                  f"{row['Benign_Pct']:<11.2f}% +{row['Difference']:.2f}%")
    else:
        print("\nNo ports significantly more common in attacks.")
    
    print("\n" + "-" * 80)
    print("PORTS MOST ASSOCIATED WITH BENIGN TRAFFIC")
    print("-" * 80)
    
    benign_distinctive = comparison_df[comparison_df['Difference'] < -5].sort_values('Difference')
    
    if len(benign_distinctive) > 0:
        print("\nPorts used significantly more in benign traffic:")
        print(f"\n{'Port':<8} {'Service':<20} {'Benign %':<12} {'Attack %':<12} {'Difference'}")
        print("-" * 80)
        for _, row in benign_distinctive.head(10).iterrows():
            print(f"{row['Port']:<8} {row['Service']:<20} {row['Benign_Pct']:<11.2f}% "
                  f"{row['Attack_Pct']:<11.2f}% {row['Difference']:.2f}%")
    else:
        print("\nNo ports significantly more common in benign traffic.")
    
    # Visualizations
    print("\n" + "=" * 80)
    print("VISUALIZATIONS")
    print("=" * 80)
    
    fig, axes = plt.subplots(2, 2, figsize=(16, 12))
    
    # 1. Side-by-side comparison of top ports
    top_compare = comparison_df.head(15)
    x = np.arange(len(top_compare))
    width = 0.35
    
    axes[0, 0].barh(x - width/2, top_compare['Benign_Pct'], width, label='Benign', color='#2ecc71')
    axes[0, 0].barh(x + width/2, top_compare['Attack_Pct'], width, label='Attack', color='#e74c3c')
    axes[0, 0].set_yticks(x)
    axes[0, 0].set_yticklabels([f"Port {row['Port']} ({row['Service']})" 
                                for _, row in top_compare.iterrows()], fontsize=8)
    axes[0, 0].set_xlabel('Percentage of Class')
    axes[0, 0].set_title(f'Port Usage by Class - {port_column}', fontweight='bold')
    axes[0, 0].legend()
    axes[0, 0].invert_yaxis()
    axes[0, 0].grid(True, alpha=0.3, axis='x')
    
    # 2. Difference plot (attack % - benign %)
    top_diff = comparison_df.nlargest(20, 'Difference', keep='all')
    colors = ['#e74c3c' if d > 0 else '#2ecc71' for d in top_diff['Difference']]
    
    axes[0, 1].barh(range(len(top_diff)), top_diff['Difference'], color=colors)
    axes[0, 1].set_yticks(range(len(top_diff)))
    axes[0, 1].set_yticklabels([f"Port {row['Port']} ({row['Service']})" 
                                for _, row in top_diff.iterrows()], fontsize=8)
    axes[0, 1].set_xlabel('Attack % - Benign % (Percentage Points)')
    axes[0, 1].set_title('Port Preference by Class\n(Positive = More in Attacks)', fontweight='bold')
    axes[0, 1].axvline(0, color='black', linewidth=1)
    axes[0, 1].invert_yaxis()
    axes[0, 1].grid(True, alpha=0.3, axis='x')
    
    # 3. Top attack ports
    top_attack = comparison_df.nlargest(10, 'Attack_Count')
    axes[1, 0].bar(range(len(top_attack)), top_attack['Attack_Count'], color='#e74c3c')
    axes[1, 0].set_xticks(range(len(top_attack)))
    axes[1, 0].set_xticklabels([f"Port {row['Port']}\n{row['Service'][:10]}" 
                                for _, row in top_attack.iterrows()], 
                               rotation=45, ha='right', fontsize=8)
    axes[1, 0].set_ylabel('Count in Attack Traffic')
    axes[1, 0].set_title('Most Common Ports in Attack Traffic', fontweight='bold')
    axes[1, 0].grid(True, alpha=0.3, axis='y')
    
    # 4. Top benign ports
    top_benign = comparison_df.nlargest(10, 'Benign_Count')
    axes[1, 1].bar(range(len(top_benign)), top_benign['Benign_Count'], color='#2ecc71')
    axes[1, 1].set_xticks(range(len(top_benign)))
    axes[1, 1].set_xticklabels([f"Port {row['Port']}\n{row['Service'][:10]}" 
                                for _, row in top_benign.iterrows()], 
                               rotation=45, ha='right', fontsize=8)
    axes[1, 1].set_ylabel('Count in Benign Traffic')
    axes[1, 1].set_title('Most Common Ports in Benign Traffic', fontweight='bold')
    axes[1, 1].grid(True, alpha=0.3, axis='y')
    
    plt.tight_layout()
    plt.show()
    
    print("\n" + "=" * 80)
    
    return comparison_df

# Compare ports by class
if 'df_with_ports' in locals():
    print("\n" + "#" * 80)
    print("# COMPARING SOURCE PORTS BY CLASS")
    print("#" * 80 + "\n")
    src_comparison = compare_ports_by_class(df_with_ports, 'Source Port', 'Binary_Label', top_n=20)
    
    print("\n\n" + "#" * 80)
    print("# COMPARING DESTINATION PORTS BY CLASS")
    print("#" * 80 + "\n")
    dst_comparison = compare_ports_by_class(df_with_ports, 'Destination Port', 'Binary_Label', top_n=20)

################################################################################
# COMPARING SOURCE PORTS BY CLASS
################################################################################

================================================================================
PORT COMPARISON BY CLASS: Source Port
================================================================================

Class 0 (BENIGN): 97,718 samples
Class 1 (ATTACK): 128,027 samples

--------------------------------------------------------------------------------
TOP PORTS BY CLASS
--------------------------------------------------------------------------------

Top ports (sorted by total occurrences):

--------------------------------------------------------------------------------
PORTS MOST ASSOCIATED WITH ATTACKS
--------------------------------------------------------------------------------

No ports significantly more common in attacks.

--------------------------------------------------------------------------------
PORTS MOST ASSOCIATED WITH BENIGN TRAFFIC
--------------------------------------------------------------------------------

Ports used significantly more in benign traffic:

Port     Service              Benign %     Attack %     Difference
--------------------------------------------------------------------------------
80       HTTP                 34.31      % 0.00       % -34.31%
443      HTTPS                6.57       % 0.00       % -6.57%

================================================================================
VISUALIZATIONS
================================================================================

================================================================================


################################################################################
# COMPARING DESTINATION PORTS BY CLASS
################################################################################

================================================================================
PORT COMPARISON BY CLASS: Destination Port
================================================================================

Class 0 (BENIGN): 97,718 samples
Class 1 (ATTACK): 128,027 samples

--------------------------------------------------------------------------------
TOP PORTS BY CLASS
--------------------------------------------------------------------------------

Top ports (sorted by total occurrences):

--------------------------------------------------------------------------------
PORTS MOST ASSOCIATED WITH ATTACKS
--------------------------------------------------------------------------------

Ports used significantly more in attacks:

Port     Service              Attack %     Benign %     Difference
--------------------------------------------------------------------------------
80       HTTP                 100.00     % 9.14       % +90.86%

--------------------------------------------------------------------------------
PORTS MOST ASSOCIATED WITH BENIGN TRAFFIC
--------------------------------------------------------------------------------

Ports used significantly more in benign traffic:

Port     Service              Benign %     Attack %     Difference
--------------------------------------------------------------------------------
53       DNS                  32.70      % 0.00       % -32.70%
443      HTTPS                13.80      % 0.00       % -13.80%

================================================================================
VISUALIZATIONS
================================================================================

================================================================================

def analyze_port_feature_importance(df, port_features, class_column='Binary_Label'):
    """
    Analyze how well port features discriminate between classes.
    """
    print("=" * 80)
    print("PORT FEATURE IMPORTANCE ANALYSIS")
    print("=" * 80)
    
    importance_data = []
    
    for feature in port_features:
        # Calculate percentage of each class that has this feature
        benign_with_feature = df[(df[class_column] == 0) & (df[feature] == 1)]
        attack_with_feature = df[(df[class_column] == 1) & (df[feature] == 1)]
        
        benign_total = (df[class_column] == 0).sum()
        attack_total = (df[class_column] == 1).sum()
        
        benign_pct = (len(benign_with_feature) / benign_total * 100) if benign_total > 0 else 0
        attack_pct = (len(attack_with_feature) / attack_total * 100) if attack_total > 0 else 0
        
        # Calculate discrimination score (absolute difference)
        discrimination = abs(attack_pct - benign_pct)
        
        importance_data.append({
            'Feature': feature,
            'Benign_Pct': benign_pct,
            'Attack_Pct': attack_pct,
            'Difference': attack_pct - benign_pct,
            'Discrimination': discrimination
        })
    
    importance_df = pd.DataFrame(importance_data).sort_values('Discrimination', ascending=False)
    
    print("\nTop 20 most discriminative port features:")
    print("(Higher discrimination = better for classification)\n")
    display(importance_df.head(20))
    
    # Visualize
    print("\n" + "=" * 80)
    print("VISUALIZATION")
    print("=" * 80)
    
    fig, axes = plt.subplots(1, 2, figsize=(16, 8))
    
    # Top discriminative features
    top_features = importance_df.head(15)
    
    axes[0].barh(range(len(top_features)), top_features['Discrimination'])
    axes[0].set_yticks(range(len(top_features)))
    axes[0].set_yticklabels(top_features['Feature'], fontsize=8)
    axes[0].set_xlabel('Discrimination Score (|Attack% - Benign%|)')
    axes[0].set_title('Most Discriminative Port Features', fontweight='bold')
    axes[0].invert_yaxis()
    axes[0].grid(True, alpha=0.3, axis='x')
    
    # Scatter plot: Benign % vs Attack %
    colors = ['#e74c3c' if d > 0 else '#2ecc71' for d in top_features['Difference']]
    
    axes[1].scatter(top_features['Benign_Pct'], top_features['Attack_Pct'], 
                   c=colors, s=100, alpha=0.6, edgecolors='black')
    axes[1].plot([0, max(top_features['Benign_Pct'].max(), top_features['Attack_Pct'].max())],
                [0, max(top_features['Benign_Pct'].max(), top_features['Attack_Pct'].max())],
                'k--', alpha=0.3, label='Equal usage')
    
    # Add labels to interesting points
    for _, row in top_features.head(10).iterrows():
        axes[1].annotate(row['Feature'].split('_')[-1][:8], 
                        (row['Benign_Pct'], row['Attack_Pct']),
                        fontsize=7, alpha=0.7)
    
    axes[1].set_xlabel('% in Benign Traffic')
    axes[1].set_ylabel('% in Attack Traffic')
    axes[1].set_title('Port Feature Usage by Class', fontweight='bold')
    axes[1].legend()
    axes[1].grid(True, alpha=0.3)
    
    plt.tight_layout()
    plt.show()
    
    print("\n💡 INTERPRETATION:")
    print("  • Features above diagonal (red): More common in attacks")
    print("  • Features below diagonal (green): More common in benign traffic")
    print("  • Features far from diagonal: Highly discriminative (useful for ML)")
    print("  • Features near diagonal: Similar usage in both classes (less useful)")
    
    print("\n" + "=" * 80)
    
    return importance_df

# Analyze feature importance
if 'df_with_ports' in locals() and 'all_port_features' in locals():
    port_importance = analyze_port_feature_importance(df_with_ports, all_port_features, 'Binary_Label')

================================================================================
PORT FEATURE IMPORTANCE ANALYSIS
================================================================================

Top 20 most discriminative port features:
(Higher discrimination = better for classification)

================================================================================
VISUALIZATION
================================================================================

💡 INTERPRETATION:
  • Features above diagonal (red): More common in attacks
  • Features below diagonal (green): More common in benign traffic
  • Features far from diagonal: Highly discriminative (useful for ML)
  • Features near diagonal: Similar usage in both classes (less useful)

================================================================================

print("=" * 80)
print("PORT FEATURE ENGINEERING SUMMARY")
print("=" * 80)

if 'df_with_ports' in locals():
    print("\n✓ Successfully created port-based features")
    print(f"\nDataset Information:")
    print(f"  Total samples: {len(df_with_ports):,}")
    print(f"  Original features: {df_binary.shape[1]}")
    print(f"  New port features: {len(all_port_features)}")
    print(f"  Total features: {df_with_ports.shape[1]}")
    
    print("\n" + "-" * 80)
    print("KEY FINDINGS")
    print("-" * 80)
    print("""
Based on the analysis above, you should have discovered:

1. PORT DISTRIBUTION:
   - Which ports are most common in your dataset
   - Distribution across well-known, registered, and dynamic port ranges
   - Which important security-relevant ports appear

2. CLASS DIFFERENCES:
   - Ports preferentially used by attackers vs benign traffic
   - Services being targeted (e.g., web servers, DNS, etc.)
   - Attack patterns visible through port selection

3. FEATURE ENGINEERING:
   - Created binary features for important ports
   - Grouped rare/unimportant ports into 'other' category
   - Reduced dimensionality while preserving discriminative information

4. FEATURE IMPORTANCE:
   - Identified most discriminative port features
   - These features should be strong predictors for classification
   - Can guide feature selection for ML models
    """)
    
    print("-" * 80)
    print("NEXT STEPS FOR SUPERVISED LEARNING")
    print("-" * 80)
    print("""
1. FEATURE SELECTION:
   - Consider using only highly discriminative port features
   - Combine with other important features from correlation analysis
   - May want to drop original Source/Destination Port columns (replaced by one-hot)

2. ADDITIONAL FEATURE ENGINEERING:
   - Could apply similar one-hot encoding to Protocol field
   - Consider interaction features (e.g., src_port_X AND dst_port_Y)
   - Time-based features if timestamps are meaningful

3. PREPROCESSING FOR ML:
   - Remove non-feature columns (IDs, IPs, timestamps, original labels)
   - Scale remaining numerical features
   - Handle any remaining missing/infinite values
   - Create train-test split with stratification

4. MODEL TRAINING:
   - Start with baseline models (Logistic Regression, Random Forest)
   - Apply class imbalance handling techniques
   - Evaluate with appropriate metrics (not just accuracy!)
   - Iterate and improve
    """)
    
    print("=" * 80)
    print("\n💾 Optional: Save dataset with port features")
    print("\n# Uncomment to save:")
    print("# df_with_ports.to_csv('network_traffic_with_port_features.csv', index=False)")
    
else:
    print("\n⚠️ Port feature engineering not completed.")
    print("   Please run the cells above first.")

print("\n" + "=" * 80)

================================================================================
PORT FEATURE ENGINEERING SUMMARY
================================================================================

✓ Successfully created port-based features

Dataset Information:
  Total samples: 225,745
  Original features: 86
  New port features: 17
  Total features: 103

--------------------------------------------------------------------------------
KEY FINDINGS
--------------------------------------------------------------------------------

Based on the analysis above, you should have discovered:

1. PORT DISTRIBUTION:
   - Which ports are most common in your dataset
   - Distribution across well-known, registered, and dynamic port ranges
   - Which important security-relevant ports appear

2. CLASS DIFFERENCES:
   - Ports preferentially used by attackers vs benign traffic
   - Services being targeted (e.g., web servers, DNS, etc.)
   - Attack patterns visible through port selection

3. FEATURE ENGINEERING:
   - Created binary features for important ports
   - Grouped rare/unimportant ports into 'other' category
   - Reduced dimensionality while preserving discriminative information

4. FEATURE IMPORTANCE:
   - Identified most discriminative port features
   - These features should be strong predictors for classification
   - Can guide feature selection for ML models
    
--------------------------------------------------------------------------------
NEXT STEPS FOR SUPERVISED LEARNING
--------------------------------------------------------------------------------

1. FEATURE SELECTION:
   - Consider using only highly discriminative port features
   - Combine with other important features from correlation analysis
   - May want to drop original Source/Destination Port columns (replaced by one-hot)

2. ADDITIONAL FEATURE ENGINEERING:
   - Could apply similar one-hot encoding to Protocol field
   - Consider interaction features (e.g., src_port_X AND dst_port_Y)
   - Time-based features if timestamps are meaningful

3. PREPROCESSING FOR ML:
   - Remove non-feature columns (IDs, IPs, timestamps, original labels)
   - Scale remaining numerical features
   - Handle any remaining missing/infinite values
   - Create train-test split with stratification

4. MODEL TRAINING:
   - Start with baseline models (Logistic Regression, Random Forest)
   - Apply class imbalance handling techniques
   - Evaluate with appropriate metrics (not just accuracy!)
   - Iterate and improve
    
================================================================================

💾 Optional: Save dataset with port features

# Uncomment to save:
# df_with_ports.to_csv('network_traffic_with_port_features.csv', index=False)

================================================================================

def plot_feature_distributions_by_class(df, features, class_column='Binary_Label', 
                                       bins=50, figsize=(20, 4)):
    """
    Plot histogram comparisons of features between classes.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset with features and class labels
    features : list
        List of feature names to plot
    class_column : str
        Name of binary class column
    bins : int
        Number of histogram bins
    figsize : tuple
        Figure size per row
    """
    n_features = len(features)
    n_cols = min(4, n_features)
    n_rows = (n_features - 1) // n_cols + 1
    
    fig, axes = plt.subplots(n_rows, n_cols, figsize=(figsize[0], figsize[1] * n_rows))
    axes = axes.flatten() if n_rows * n_cols > 1 else [axes]
    
    # Separate by class
    benign = df[df[class_column] == 0]
    attack = df[df[class_column] == 1]
    
    for idx, feature in enumerate(features):
        if feature not in df.columns:
            axes[idx].text(0.5, 0.5, f'Feature not found:\n{feature}', 
                          ha='center', va='center', transform=axes[idx].transAxes)
            continue
        
        # Get data and clean
        benign_data = benign[feature].replace([np.inf, -np.inf], np.nan).dropna()
        attack_data = attack[feature].replace([np.inf, -np.inf], np.nan).dropna()
        
        if len(benign_data) == 0 and len(attack_data) == 0:
            axes[idx].text(0.5, 0.5, f'No data for:\n{feature}', 
                          ha='center', va='center', transform=axes[idx].transAxes)
            continue
        
        # Determine if we need log scale (high skew)
        all_data = pd.concat([benign_data, attack_data])
        use_log = (all_data.max() / all_data.median() > 100) if all_data.median() > 0 else False
        
        # Plot overlaid histograms
        if use_log:
            # Use log scale
            benign_data_plot = benign_data[benign_data > 0]
            attack_data_plot = attack_data[attack_data > 0]
            
            if len(benign_data_plot) > 0:
                axes[idx].hist(benign_data_plot, bins=bins, alpha=0.6, 
                              label=f'Benign (n={len(benign_data_plot):,})',
                              color='#2ecc71', edgecolor='black', log=True)
            if len(attack_data_plot) > 0:
                axes[idx].hist(attack_data_plot, bins=bins, alpha=0.6,
                              label=f'Attack (n={len(attack_data_plot):,})',
                              color='#e74c3c', edgecolor='black', log=True)
            axes[idx].set_ylabel('Frequency (log scale)')
        else:
            # Regular scale
            axes[idx].hist(benign_data, bins=bins, alpha=0.6,
                          label=f'Benign (n={len(benign_data):,})',
                          color='#2ecc71', edgecolor='black')
            axes[idx].hist(attack_data, bins=bins, alpha=0.6,
                          label=f'Attack (n={len(attack_data):,})',
                          color='#e74c3c', edgecolor='black')
            axes[idx].set_ylabel('Frequency')
        
        # Add statistics
        benign_mean = benign_data.mean()
        attack_mean = attack_data.mean()
        benign_median = benign_data.median()
        attack_median = attack_data.median()
        
        # Add mean lines
        axes[idx].axvline(benign_mean, color='#27ae60', linestyle='--', linewidth=2, alpha=0.8)
        axes[idx].axvline(attack_mean, color='#c0392b', linestyle='--', linewidth=2, alpha=0.8)
        
        axes[idx].set_xlabel('Value')
        axes[idx].set_title(f'{feature}\nBenign μ={benign_mean:.2f}, Attack μ={attack_mean:.2f}',
                           fontsize=10, fontweight='bold')
        axes[idx].legend(loc='upper right', fontsize=8)
        axes[idx].grid(True, alpha=0.3)
    
    # Hide unused subplots
    for idx in range(len(features), len(axes)):
        axes[idx].axis('off')
    
    plt.tight_layout()
    plt.show()

print("Function loaded: plot_feature_distributions_by_class()")

Function loaded: plot_feature_distributions_by_class()

if 'df_with_ports' in locals():
    print("=" * 80)
    print("BYTES FEATURES COMPARISON")
    print("=" * 80)
    
    bytes_features = [
        'Total Length of Fwd Packets',
        'Total Length of Bwd Packets',
        'Fwd Packet Length Mean',
        'Bwd Packet Length Mean'
    ]
    
    plot_feature_distributions_by_class(df_with_ports, bytes_features, 'Binary_Label')
    
    # Statistical summary
    print("\nStatistical Summary:")
    print("-" * 80)
    
    benign = df_with_ports[df_with_ports['Binary_Label'] == 0]
    attack = df_with_ports[df_with_ports['Binary_Label'] == 1]
    
    summary_data = []
    for feature in bytes_features:
        if feature in df_with_ports.columns:
            benign_data = benign[feature].replace([np.inf, -np.inf], np.nan).dropna()
            attack_data = attack[feature].replace([np.inf, -np.inf], np.nan).dropna()
            
            summary_data.append({
                'Feature': feature,
                'Benign_Mean': benign_data.mean(),
                'Attack_Mean': attack_data.mean(),
                'Benign_Median': benign_data.median(),
                'Attack_Median': attack_data.median(),
                'Difference_Mean': attack_data.mean() - benign_data.mean()
            })
    
    summary_df = pd.DataFrame(summary_data)
    display(summary_df)
    
    print("\n💡 INTERPRETATION:")
    print("  • Higher values in one class indicate different data transfer patterns")
    print("  • Large differences suggest good discriminative features")

================================================================================
BYTES FEATURES COMPARISON
================================================================================

Statistical Summary:
--------------------------------------------------------------------------------

💡 INTERPRETATION:
  • Higher values in one class indicate different data transfer patterns
  • Large differences suggest good discriminative features

if 'df_with_ports' in locals():
    print("=" * 80)
    print("PACKETS FEATURES COMPARISON")
    print("=" * 80)
    
    packets_features = [
        'Total Fwd Packets',
        'Total Backward Packets',
        'Flow Packets/s',
        'Fwd Packets/s'
    ]
    
    plot_feature_distributions_by_class(df_with_ports, packets_features, 'Binary_Label')
    
    # Statistical summary
    print("\nStatistical Summary:")
    print("-" * 80)
    
    benign = df_with_ports[df_with_ports['Binary_Label'] == 0]
    attack = df_with_ports[df_with_ports['Binary_Label'] == 1]
    
    summary_data = []
    for feature in packets_features:
        if feature in df_with_ports.columns:
            benign_data = benign[feature].replace([np.inf, -np.inf], np.nan).dropna()
            attack_data = attack[feature].replace([np.inf, -np.inf], np.nan).dropna()
            
            summary_data.append({
                'Feature': feature,
                'Benign_Mean': benign_data.mean(),
                'Attack_Mean': attack_data.mean(),
                'Benign_Median': benign_data.median(),
                'Attack_Median': attack_data.median(),
                'Difference_Mean': attack_data.mean() - benign_data.mean()
            })
    
    summary_df = pd.DataFrame(summary_data)
    display(summary_df)
    
    print("\n💡 INTERPRETATION:")
    print("  • Attack traffic may show higher packet rates (flooding behavior)")
    print("  • Asymmetric packet counts (fwd >> bwd) indicate one-way flood")

================================================================================
PACKETS FEATURES COMPARISON
================================================================================

Statistical Summary:
--------------------------------------------------------------------------------

💡 INTERPRETATION:
  • Attack traffic may show higher packet rates (flooding behavior)
  • Asymmetric packet counts (fwd >> bwd) indicate one-way flood

if 'df_with_ports' in locals():
    print("=" * 80)
    print("DURATION FEATURES COMPARISON")
    print("=" * 80)
    
    duration_features = [
        'Flow Duration',
        'Flow IAT Mean',
        'Fwd IAT Mean',
        'Bwd IAT Mean'
    ]
    
    plot_feature_distributions_by_class(df_with_ports, duration_features, 'Binary_Label')
    
    # Statistical summary
    print("\nStatistical Summary:")
    print("-" * 80)
    
    benign = df_with_ports[df_with_ports['Binary_Label'] == 0]
    attack = df_with_ports[df_with_ports['Binary_Label'] == 1]
    
    summary_data = []
    for feature in duration_features:
        if feature in df_with_ports.columns:
            benign_data = benign[feature].replace([np.inf, -np.inf], np.nan).dropna()
            attack_data = attack[feature].replace([np.inf, -np.inf], np.nan).dropna()
            
            # Convert to seconds for readability
            benign_mean_sec = benign_data.mean() / 1000000  # microseconds to seconds
            attack_mean_sec = attack_data.mean() / 1000000
            
            summary_data.append({
                'Feature': feature,
                'Benign_Mean_sec': benign_mean_sec,
                'Attack_Mean_sec': attack_mean_sec,
                'Benign_Median_sec': benign_data.median() / 1000000,
                'Attack_Median_sec': attack_data.median() / 1000000,
                'Difference_sec': attack_mean_sec - benign_mean_sec
            })
    
    summary_df = pd.DataFrame(summary_data)
    display(summary_df)
    
    print("\n💡 INTERPRETATION:")
    print("  •Flow Duration: Total time of the connection")
    print("  • IAT (Inter-Arrival Time): Time between packets")
    print("  • Shorter durations or IAT may indicate rapid flooding behavior")

================================================================================
DURATION FEATURES COMPARISON
================================================================================

Statistical Summary:
--------------------------------------------------------------------------------

💡 INTERPRETATION:
  •Flow Duration: Total time of the connection
  • IAT (Inter-Arrival Time): Time between packets
  • Shorter durations or IAT may indicate rapid flooding behavior

def plot_boxplot_comparison(df, features, class_column='Binary_Label'):
    """
    Create side-by-side box plots for feature comparison.
    """
    n_features = len(features)
    n_cols = 4
    n_rows = (n_features - 1) // n_cols + 1
    
    fig, axes = plt.subplots(n_rows, n_cols, figsize=(20, 5 * n_rows))
    axes = axes.flatten() if n_rows * n_cols > 1 else [axes]
    
    for idx, feature in enumerate(features):
        if feature not in df.columns:
            axes[idx].text(0.5, 0.5, f'Feature not found', 
                          ha='center', va='center', transform=axes[idx].transAxes)
            continue
        
        # Prepare data
        plot_data = []
        labels = []
        
        for class_val, class_name, color in [(0, 'Benign', '#2ecc71'), (1, 'Attack', '#e74c3c')]:
            data = df[df[class_column] == class_val][feature]
            data = data.replace([np.inf, -np.inf], np.nan).dropna()
            
            if len(data) > 0:
                plot_data.append(data)
                labels.append(f'{class_name}\n(n={len(data):,})')
        
        if len(plot_data) > 0:
            bp = axes[idx].boxplot(plot_data, labels=labels, patch_artist=True,
                                   showfliers=False)  # Hide outliers for clarity
            
            # Color the boxes
            colors = ['#2ecc71', '#e74c3c'][:len(plot_data)]
            for patch, color in zip(bp['boxes'], colors):
                patch.set_facecolor(color)
                patch.set_alpha(0.6)
        
        axes[idx].set_title(feature, fontsize=10, fontweight='bold')
        axes[idx].set_ylabel('Value')
        axes[idx].grid(True, alpha=0.3, axis='y')
    
    # Hide unused subplots
    for idx in range(len(features), len(axes)):
        axes[idx].axis('off')
    
    plt.suptitle('Feature Distribution Comparison: Benign vs Attack', 
                fontsize=16, fontweight='bold', y=1.00)
    plt.tight_layout()
    plt.show()

if 'df_with_ports' in locals():
    print("=" * 80)
    print("BOX PLOT COMPARISON - KEY FEATURES")
    print("=" * 80)
    
    key_features = [
        'Flow Duration',
        'Total Fwd Packets',
        'Total Backward Packets',
        'Total Length of Fwd Packets',
        'Total Length of Bwd Packets',
        'Flow Bytes/s',
        'Flow Packets/s',
        'Fwd Packet Length Mean'
    ]
    
    plot_boxplot_comparison(df_with_ports, key_features, 'Binary_Label')
    
    print("\n💡 BOX PLOT INTERPRETATION:")
    print("  • Box: Middle 50% of data (IQR)")
    print("  • Line in box: Median")
    print("  • Whiskers: Extend to 1.5 × IQR")
    print("  • Different box positions = different distributions between classes")

================================================================================
BOX PLOT COMPARISON - KEY FEATURES
================================================================================

💡 BOX PLOT INTERPRETATION:
  • Box: Middle 50% of data (IQR)
  • Line in box: Median
  • Whiskers: Extend to 1.5 × IQR
  • Different box positions = different distributions between classes

def calculate_feature_discrimination(df, features, class_column='Binary_Label'):
    """
    Calculate discrimination metrics for features.
    """
    print("=" * 80)
    print("FEATURE DISCRIMINATION ANALYSIS")
    print("=" * 80)
    
    benign = df[df[class_column] == 0]
    attack = df[df[class_column] == 1]
    
    discrimination_data = []
    
    for feature in features:
        if feature not in df.columns:
            continue
        
        benign_data = benign[feature].replace([np.inf, -np.inf], np.nan).dropna()
        attack_data = attack[feature].replace([np.inf, -np.inf], np.nan).dropna()
        
        if len(benign_data) == 0 or len(attack_data) == 0:
            continue
        
        # Calculate metrics
        benign_mean = benign_data.mean()
        attack_mean = attack_data.mean()
        benign_std = benign_data.std()
        attack_std = attack_data.std()
        
        # Normalized difference (Cohen's d - effect size)
        pooled_std = np.sqrt((benign_std**2 + attack_std**2) / 2)
        cohens_d = abs(attack_mean - benign_mean) / pooled_std if pooled_std > 0 else 0
        
        # Separation ratio
        separation = abs(attack_mean - benign_mean) / max(benign_mean, attack_mean) if max(benign_mean, attack_mean) > 0 else 0
        
        discrimination_data.append({
            'Feature': feature,
            'Benign_Mean': benign_mean,
            'Attack_Mean': attack_mean,
            'Abs_Difference': abs(attack_mean - benign_mean),
            'Cohens_d': cohens_d,
            'Separation_Ratio': separation
        })
    
    discrimination_df = pd.DataFrame(discrimination_data)
    
    print("\nTop 15 Most Discriminative Features (by Cohen's d):")
    print("(Cohen's d > 0.8 = large effect, > 0.5 = medium, > 0.2 = small)\n")
    
    top_features = discrimination_df.nlargest(15, 'Cohens_d')
    display(top_features)
    
    # Visualization
    plt.figure(figsize=(14, 8))
    
    top_15 = discrimination_df.nlargest(15, 'Cohens_d')
    
    plt.barh(range(len(top_15)), top_15['Cohens_d'])
    plt.yticks(range(len(top_15)), top_15['Feature'], fontsize=9)
    plt.xlabel("Cohen's d (Effect Size)", fontsize=12)
    plt.title('Most Discriminative Features for Classification\n(Higher = Better Separation)', 
             fontsize=14, fontweight='bold')
    plt.axvline(0.2, color='orange', linestyle='--', alpha=0.5, label='Small effect')
    plt.axvline(0.5, color='yellow', linestyle='--', alpha=0.5, label='Medium effect')
    plt.axvline(0.8, color='red', linestyle='--', alpha=0.5, label='Large effect')
    plt.legend()
    plt.grid(True, alpha=0.3, axis='x')
    plt.tight_layout()
    plt.show()
    
    print("\n" + "=" * 80)
    
    return discrimination_df

if 'df_with_ports' in locals():
    # Analyze all numeric features
    numeric_features = df_with_ports.select_dtypes(include=[np.number]).columns.tolist()
    
    # Remove label and ID columns
    exclude_cols = ['Binary_Label', 'Label'] + [col for col in numeric_features if 'port_' in col.lower()]
    analysis_features = [f for f in numeric_features if f not in exclude_cols]
    
    feature_discrimination = calculate_feature_discrimination(
        df_with_ports, 
        analysis_features, 
        'Binary_Label'
    )

================================================================================
FEATURE DISCRIMINATION ANALYSIS
================================================================================

Top 15 Most Discriminative Features (by Cohen's d):
(Cohen's d > 0.8 = large effect, > 0.5 = medium, > 0.2 = small)

================================================================================

# Import machine learning libraries
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score, 
    roc_auc_score, confusion_matrix, classification_report,
    make_scorer
)
import xgboost as xgb
from sklearn.svm import SVC

def prepare_features_for_ml(df, target_column='Binary_Label'):
    """
    Prepare final feature matrix for machine learning.
    
    Parameters:
    -----------
    df : pandas DataFrame
        Dataset with all features and target
    target_column : str
        Name of target variable column
    
    Returns:
    --------
    X : pandas DataFrame
        Feature matrix
    y : pandas Series
        Target variable
    feature_names : list
        List of feature names
    """
    print("=" * 80)
    print("FEATURE PREPARATION FOR MACHINE LEARNING")
    print("=" * 80)
    
    df_ml = df.copy()
    
    # 1. Remove non-feature columns
    non_feature_cols = [
        'Flow ID', 
        'Source IP', 
        'Destination IP', 
        'Timestamp',
        'Label',  # Original multi-class label
        target_column  # Binary label (we'll extract this separately)
    ]
    
    # Also remove original port columns if we created one-hot encoded versions
    port_cols = ['Source Port', 'Destination Port']
    
    # Check which columns actually exist
    cols_to_remove = [col for col in non_feature_cols + port_cols if col in df_ml.columns]
    
    print(f"\n1. Removing {len(cols_to_remove)} non-feature columns:")
    for col in cols_to_remove:
        print(f"   - {col}")
    
    # Extract target before removing columns
    y = df_ml[target_column].copy()
    
    # Remove non-feature columns
    df_features = df_ml.drop(columns=cols_to_remove, errors='ignore')
    
    # 2. Ensure all features are numeric
    print(f"\n2. Ensuring all features are numeric...")
    non_numeric = df_features.select_dtypes(exclude=[np.number]).columns.tolist()
    if non_numeric:
        print(f"   ⚠️ Found {len(non_numeric)} non-numeric columns:")
        for col in non_numeric:
            print(f"     - {col}")
        print(f"   These will be dropped.")
        df_features = df_features.select_dtypes(include=[np.number])
    else:
        print("   ✓ All features are numeric")
    
    # 3. Handle infinite values
    print(f"\n3. Handling infinite values...")
    inf_count_before = np.isinf(df_features.values).sum()
    if inf_count_before > 0:
        print(f"   Found {inf_count_before:,} infinite values")
        df_features = df_features.replace([np.inf, -np.inf], np.nan)
        print(f"   ✓ Replaced with NaN")
    else:
        print("   ✓ No infinite values found")
    
    # 4. Handle missing values
    print(f"\n4. Handling missing values...")
    missing_count = df_features.isnull().sum().sum()
    if missing_count > 0:
        print(f"   Found {missing_count:,} missing values")
        print(f"   Imputing with column medians...")
        df_features = df_features.fillna(df_features.median())
        print(f"   ✓ Missing values imputed")
    else:
        print("   ✓ No missing values found")
    
    # Final feature set
    X = df_features
    feature_names = X.columns.tolist()
    
    print(f"\n" + "=" * 80)
    print("FINAL DATASET FOR MODELING")
    print("=" * 80)
    print(f"\nSamples: {len(X):,}")
    print(f"Features: {len(feature_names)}")
    print(f"\nClass distribution:")
    print(f"  Class 0 (BENIGN): {(y == 0).sum():,} ({(y == 0).sum() / len(y) * 100:.2f}%)")
    print(f"  Class 1 (ATTACK): {(y == 1).sum():,} ({(y == 1).sum() / len(y) * 100:.2f}%)")
    
    print(f"\nFeature types:")
    # Count port features vs other features
    port_features = [f for f in feature_names if 'port_' in f.lower()]
    other_features = [f for f in feature_names if 'port_' not in f.lower()]
    print(f"  Port features: {len(port_features)}")
    print(f"  Other features: {len(other_features)}")
    
    # Sample feature names
    print(f"\nSample feature names:")
    for i, feat in enumerate(feature_names[:10], 1):
        print(f"  {i}. {feat}")
    if len(feature_names) > 10:
        print(f"  ... and {len(feature_names) - 10} more")
    
    print("\n" + "=" * 80)
    
    return X, y, feature_names

# Prepare features
if 'df_with_ports' in locals():
    X, y, feature_names = prepare_features_for_ml(df_with_ports, target_column='Binary_Label')
    print("\n✓ Features prepared successfully!")
else:
    print("⚠️ Warning: 'df_with_ports' not found. Please run previous cells first.")

================================================================================
FEATURE PREPARATION FOR MACHINE LEARNING
================================================================================

1. Removing 8 non-feature columns:
   - Flow ID
   - Source IP
   - Destination IP
   - Timestamp
   - Label
   - Binary_Label
   - Source Port
   - Destination Port

2. Ensuring all features are numeric...
   ✓ All features are numeric

3. Handling infinite values...
   Found 64 infinite values
   ✓ Replaced with NaN

4. Handling missing values...
   Found 68 missing values
   Imputing with column medians...
   ✓ Missing values imputed

================================================================================
FINAL DATASET FOR MODELING
================================================================================

Samples: 225,745
Features: 95

Class distribution:
  Class 0 (BENIGN): 97,718 (43.29%)
  Class 1 (ATTACK): 128,027 (56.71%)

Feature types:
  Port features: 17
  Other features: 78

Sample feature names:
  1. Protocol
  2. Flow Duration
  3. Total Fwd Packets
  4. Total Backward Packets
  5. Total Length of Fwd Packets
  6. Total Length of Bwd Packets
  7. Fwd Packet Length Max
  8. Fwd Packet Length Min
  9. Fwd Packet Length Mean
  10. Fwd Packet Length Std
  ... and 85 more

================================================================================

✓ Features prepared successfully!

# Define scoring metrics for cross-validation
scoring = {
    'accuracy': 'accuracy',
    'precision': make_scorer(precision_score, zero_division=0),
    'recall': make_scorer(recall_score, zero_division=0),
    'f1': make_scorer(f1_score, zero_division=0),
    'roc_auc': 'roc_auc'
}

# Setup stratified K-fold
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

print("=" * 80)
print("CROSS-VALIDATION SETUP")
print("=" * 80)
print("\n✓ Configuration:")
print(f"  Method: Stratified K-Fold")
print(f"  Number of folds: 10")
print(f"  Shuffle: Yes (random_state=42)")
print(f"\n✓ Metrics to track: {list(scoring.keys())}")
print("\n✓ Ready for model training!")
print("=" * 80)

================================================================================
CROSS-VALIDATION SETUP
================================================================================

✓ Configuration:
  Method: Stratified K-Fold
  Number of folds: 10
  Shuffle: Yes (random_state=42)

✓ Metrics to track: ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']

✓ Ready for model training!
================================================================================

import time
import numpy as np
from sklearn.model_selection import train_test_split

print("="*80)
print("COMPUTATIONAL FEASIBILITY ANALYSIS")
print("="*80)
X_ml = X
y_ml = y
# Create stratified sample for testing
X_sample, _, y_sample, _ = train_test_split(
    X_ml, y_ml, 
    train_size=10000,
    stratify=y_ml,
    random_state=42
)

print(f"\nTesting on {len(X_sample):,} samples ({len(X_sample)/len(X_ml)*100:.1f}% of full dataset)")
print(f"Full dataset size: {len(X_ml):,} samples\n")

models_to_test = {
    'Logistic Regression': LogisticRegression(max_iter=1000, random_state=42),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'XGBoost': xgb.XGBClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'SVM (RBF)': SVC(kernel='rbf', random_state=42)
}

timing_results = []

for model_name, model in models_to_test.items():
    print(f"Testing {model_name}...")
    
    # Time single training run on sample
    start_time = time.time()
    model.fit(X_sample, y_sample)
    elapsed = time.time() - start_time
    
    # Project to full dataset (rough estimate based on complexity)
    if model_name == 'SVM (RBF)':
        # SVM scales roughly O(n^2) to O(n^3)
        scale_factor = (len(X_ml) / len(X_sample)) ** 2.5  # Conservative estimate
    else:
        # Most tree models scale roughly O(n log n)
        scale_factor = (len(X_ml) / len(X_sample)) * np.log(len(X_ml)) / np.log(len(X_sample))
    
    projected_single = elapsed * scale_factor
    projected_cv = projected_single * 10  # 10-fold CV
    
    timing_results.append({
        'Model': model_name,
        'Sample_Time': elapsed,
        'Projected_Single': projected_single,
        'Projected_10fold_CV': projected_cv
    })
    
    print(f"  Sample (10k): {elapsed:.1f} seconds")
    print(f"  Projected full dataset: {projected_single/60:.1f} minutes")
    print(f"  Projected 10-fold CV: {projected_cv/3600:.1f} hours")
    print()

# Create summary table
timing_df = pd.DataFrame(timing_results)
print("="*80)
print("TIMING SUMMARY")
print("="*80)
display(timing_df)

================================================================================
COMPUTATIONAL FEASIBILITY ANALYSIS
================================================================================

Testing on 10,000 samples (4.4% of full dataset)
Full dataset size: 225,745 samples

Testing Logistic Regression...
  Sample (10k): 0.4 seconds
  Projected full dataset: 0.2 minutes
  Projected 10-fold CV: 0.0 hours

Testing Random Forest...
  Sample (10k): 0.3 seconds
  Projected full dataset: 0.1 minutes
  Projected 10-fold CV: 0.0 hours

Testing XGBoost...
  Sample (10k): 0.2 seconds
  Projected full dataset: 0.1 minutes
  Projected 10-fold CV: 0.0 hours

Testing SVM (RBF)...
  Sample (10k): 1.3 seconds
  Projected full dataset: 53.9 minutes
  Projected 10-fold CV: 9.0 hours

================================================================================
TIMING SUMMARY
================================================================================

def train_logistic_regression(X, y, cv, scoring):
    """
    Train and evaluate Logistic Regression with 10-fold CV.
    """
    print("=" * 80)
    print("MODEL 1: LOGISTIC REGRESSION")
    print("=" * 80)
    
    print("\n📊 Training with 10-fold cross-validation...")
    print("   This may take a few minutes...\n")
    
    start_time = time.time()
    
    # Scale features (important for logistic regression)
    print("1. Scaling features...")
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)
    
    # Create model
    print("2. Creating Logistic Regression model...")
    model = LogisticRegression(
        class_weight='balanced',  # Handle imbalance
        max_iter=1000,
        random_state=42,
        n_jobs=-1  # Use all CPU cores
    )
    
    # Perform cross-validation
    print("3. Performing 10-fold cross-validation...")
    cv_results = cross_validate(
        model, X_scaled, y,
        cv=cv,
        scoring=scoring,
        return_train_score=True,
        n_jobs=-1
    )
    
    elapsed = time.time() - start_time
    
    print(f"\n✓ Cross-validation complete in {elapsed:.2f} seconds")
    
    # Calculate statistics
    results = {}
    print("\n" + "=" * 80)
    print("LOGISTIC REGRESSION - 10-FOLD CV RESULTS")
    print("=" * 80)
    
    for metric_name in scoring.keys():
        test_scores = cv_results[f'test_{metric_name}']
        train_scores = cv_results[f'train_{metric_name}']
        
        results[metric_name] = {
            'test_mean': test_scores.mean(),
            'test_std': test_scores.std(),
            'train_mean': train_scores.mean(),
            'train_std': train_scores.std()
        }
        
        print(f"\n{metric_name.upper()}:")
        print(f"  Test:  {test_scores.mean():.4f} (+/- {test_scores.std():.4f})")
        print(f"  Train: {train_scores.mean():.4f} (+/- {train_scores.std():.4f})")
    
    print("\n" + "=" * 80)
    
    return results, cv_results, model, scaler

# Train Logistic Regression
if 'X' in locals() and 'y' in locals():
    lr_results, lr_cv_results, lr_model, lr_scaler = train_logistic_regression(X, y, cv, scoring)
else:
    print("⚠️ Please run feature preparation cells first")

================================================================================
MODEL 1: LOGISTIC REGRESSION
================================================================================

📊 Training with 10-fold cross-validation...
   This may take a few minutes...

1. Scaling features...
2. Creating Logistic Regression model...
3. Performing 10-fold cross-validation...

✓ Cross-validation complete in 5.84 seconds

================================================================================
LOGISTIC REGRESSION - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  0.9993 (+/- 0.0001)
  Train: 0.9994 (+/- 0.0000)

PRECISION:
  Test:  0.9997 (+/- 0.0002)
  Train: 0.9997 (+/- 0.0000)

RECALL:
  Test:  0.9992 (+/- 0.0002)
  Train: 0.9992 (+/- 0.0000)

F1:
  Test:  0.9994 (+/- 0.0001)
  Train: 0.9995 (+/- 0.0000)

ROC_AUC:
  Test:  0.9999 (+/- 0.0001)
  Train: 0.9999 (+/- 0.0000)

================================================================================

def train_random_forest(X, y, cv, scoring):
    """
    Train and evaluate Random Forest with 10-fold CV.
    """
    print("=" * 80)
    print("MODEL 2: RANDOM FOREST")
    print("=" * 80)
    
    print("\n📊 Training with 10-fold cross-validation...")
    print("   This may take several minutes...\n")
    
    start_time = time.time()
    
    # Create model (no scaling needed for tree-based models)
    print("Creating Random Forest model...")
    model = RandomForestClassifier(
        n_estimators=100,
        max_depth=20,
        class_weight='balanced',
        random_state=42,
        n_jobs=-1
    )
    
    # Perform cross-validation
    print("Performing 10-fold cross-validation...")
    cv_results = cross_validate(
        model, X, y,
        cv=cv,
        scoring=scoring,
        return_train_score=True,
        n_jobs=-1
    )
    
    elapsed = time.time() - start_time
    
    print(f"\n✓ Cross-validation complete in {elapsed:.2f} seconds")
    
    # Calculate statistics
    results = {}
    print("\n" + "=" * 80)
    print("RANDOM FOREST - 10-FOLD CV RESULTS")
    print("=" * 80)
    
    for metric_name in scoring.keys():
        test_scores = cv_results[f'test_{metric_name}']
        train_scores = cv_results[f'train_{metric_name}']
        
        results[metric_name] = {
            'test_mean': test_scores.mean(),
            'test_std': test_scores.std(),
            'train_mean': train_scores.mean(),
            'train_std': train_scores.std()
        }
        
        print(f"\n{metric_name.upper()}:")
        print(f"  Test:  {test_scores.mean():.4f} (+/- {test_scores.std():.4f})")
        print(f"  Train: {train_scores.mean():.4f} (+/- {train_scores.std():.4f})")
    
    print("\n" + "=" * 80)
    
    return results, cv_results, model

# Train Random Forest
if 'X' in locals() and 'y' in locals():
    rf_results, rf_cv_results, rf_model = train_random_forest(X, y, cv, scoring)
else:
    print("⚠️ Please run feature preparation cells first")

================================================================================
MODEL 2: RANDOM FOREST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Creating Random Forest model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 19.48 seconds

================================================================================
RANDOM FOREST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  0.9999 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

PRECISION:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

RECALL:
  Test:  0.9999 (+/- 0.0001)
  Train: 1.0000 (+/- 0.0000)

F1:
  Test:  0.9999 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

ROC_AUC:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

================================================================================

def train_xgboost(X, y, cv, scoring):
    """
    Train and evaluate XGBoost with 10-fold CV.
    """
    print("=" * 80)
    print("MODEL 3: XGBOOST")
    print("=" * 80)
    
    print("\n📊 Training with 10-fold cross-validation...")
    print("   This may take several minutes...\n")
    
    start_time = time.time()
    
    # Calculate scale_pos_weight for imbalanced data
    neg_count = (y == 0).sum()
    pos_count = (y == 1).sum()
    scale_pos_weight = neg_count / pos_count
    
    print(f"Calculated scale_pos_weight: {scale_pos_weight:.2f}")
    print(f"  (Ratio of negative to positive samples)\n")
    
    # Create model
    print("Creating XGBoost model...")
    model = xgb.XGBClassifier(
        n_estimators=100,
        max_depth=6,
        learning_rate=0.1,
        scale_pos_weight=scale_pos_weight,
        random_state=42,
        n_jobs=-1,
        eval_metric='logloss'
    )
    
    # Perform cross-validation
    print("Performing 10-fold cross-validation...")
    cv_results = cross_validate(
        model, X, y,
        cv=cv,
        scoring=scoring,
        return_train_score=True,
        n_jobs=-1
    )
    
    elapsed = time.time() - start_time
    
    print(f"\n✓ Cross-validation complete in {elapsed:.2f} seconds")
    
    # Calculate statistics
    results = {}
    print("\n" + "=" * 80)
    print("XGBOOST - 10-FOLD CV RESULTS")
    print("=" * 80)
    
    for metric_name in scoring.keys():
        test_scores = cv_results[f'test_{metric_name}']
        train_scores = cv_results[f'train_{metric_name}']
        
        results[metric_name] = {
            'test_mean': test_scores.mean(),
            'test_std': test_scores.std(),
            'train_mean': train_scores.mean(),
            'train_std': train_scores.std()
        }
        
        print(f"\n{metric_name.upper()}:")
        print(f"  Test:  {test_scores.mean():.4f} (+/- {test_scores.std():.4f})")
        print(f"  Train: {train_scores.mean():.4f} (+/- {train_scores.std():.4f})")
    
    print("\n" + "=" * 80)
    
    return results, cv_results, model

# Train XGBoost
if 'X' in locals() and 'y' in locals():
    xgb_results, xgb_cv_results, xgb_model = train_xgboost(X, y, cv, scoring)
else:
    print("⚠️ Please run feature preparation cells first")

================================================================================
MODEL 3: XGBOOST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Calculated scale_pos_weight: 0.76
  (Ratio of negative to positive samples)

Creating XGBoost model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 5.30 seconds

================================================================================
XGBOOST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

PRECISION:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

RECALL:
  Test:  0.9999 (+/- 0.0001)
  Train: 1.0000 (+/- 0.0000)

F1:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

ROC_AUC:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

================================================================================

def compare_models(lr_results, rf_results, xgb_results):
    """
    Create comprehensive comparison of all models.
    """
    print("=" * 80)
    print("MODEL COMPARISON - 10-FOLD CV TEST SET PERFORMANCE")
    print("=" * 80)
    
    # Compile results into DataFrame
    comparison_data = []
    
    models_dict = {
        'Logistic Regression': lr_results,
        'Random Forest': rf_results,
        'XGBoost': xgb_results,
    }
    
    for model_name, results in models_dict.items():
        row = {'Model': model_name}
        for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
            row[f'{metric}_mean'] = results[metric]['test_mean']
            row[f'{metric}_std'] = results[metric]['test_std']
        comparison_data.append(row)
    
    comparison_df = pd.DataFrame(comparison_data)
    
    # Display formatted results
    print("\n📊 MEAN PERFORMANCE (± STD)\n")
    
    for _, row in comparison_df.iterrows():
        print(f"\n{row['Model']}:")
        print("-" * 80)
        print(f"  Accuracy:  {row['accuracy_mean']:.4f} (± {row['accuracy_std']:.4f})")
        print(f"  Precision: {row['precision_mean']:.4f} (± {row['precision_std']:.4f})")
        print(f"  Recall:    {row['recall_mean']:.4f} (± {row['recall_std']:.4f})")
        print(f"  F1-Score:  {row['f1_mean']:.4f} (± {row['f1_std']:.4f})")
        print(f"  ROC-AUC:   {row['roc_auc_mean']:.4f} (± {row['roc_auc_std']:.4f})")
    
    # Highlight best models for each metric
    print("\n" + "=" * 80)
    print("BEST MODEL BY METRIC")
    print("=" * 80)
    
    for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
        best_idx = comparison_df[f'{metric}_mean'].idxmax()
        best_model = comparison_df.loc[best_idx, 'Model']
        best_score = comparison_df.loc[best_idx, f'{metric}_mean']
        best_std = comparison_df.loc[best_idx, f'{metric}_std']
        
        print(f"\n{metric.upper()}:")
        print(f"  🏆 {best_model}: {best_score:.4f} (± {best_std:.4f})")
    
    print("\n" + "=" * 80)
    
    return comparison_df

# Compare all models
# Compare all models
try:
    comparison_df = compare_models(lr_results, rf_results, xgb_results)
except NameError as e:
    print(f"⚠️ Please train all models first")
    print(f"   Missing: {e}")

================================================================================
MODEL COMPARISON - 10-FOLD CV TEST SET PERFORMANCE
================================================================================

📊 MEAN PERFORMANCE (± STD)


Logistic Regression:
--------------------------------------------------------------------------------
  Accuracy:  0.9993 (± 0.0001)
  Precision: 0.9997 (± 0.0002)
  Recall:    0.9992 (± 0.0002)
  F1-Score:  0.9994 (± 0.0001)
  ROC-AUC:   0.9999 (± 0.0001)

Random Forest:
--------------------------------------------------------------------------------
  Accuracy:  0.9999 (± 0.0000)
  Precision: 1.0000 (± 0.0000)
  Recall:    0.9999 (± 0.0001)
  F1-Score:  0.9999 (± 0.0000)
  ROC-AUC:   1.0000 (± 0.0000)

XGBoost:
--------------------------------------------------------------------------------
  Accuracy:  1.0000 (± 0.0000)
  Precision: 1.0000 (± 0.0000)
  Recall:    0.9999 (± 0.0001)
  F1-Score:  1.0000 (± 0.0000)
  ROC-AUC:   1.0000 (± 0.0000)

================================================================================
BEST MODEL BY METRIC
================================================================================

ACCURACY:
  🏆 XGBoost: 1.0000 (± 0.0000)

PRECISION:
  🏆 Random Forest: 1.0000 (± 0.0000)

RECALL:
  🏆 XGBoost: 0.9999 (± 0.0001)

F1:
  🏆 XGBoost: 1.0000 (± 0.0000)

ROC_AUC:
  🏆 XGBoost: 1.0000 (± 0.0000)

================================================================================

def visualize_model_comparison(comparison_df):
    """
    Create visualizations comparing all models.
    """
    print("=" * 80)
    print("MODEL COMPARISON VISUALIZATIONS")
    print("=" * 80)
    
    # Create subplots
    fig, axes = plt.subplots(2, 3, figsize=(18, 10))
    axes = axes.flatten()
    
    metrics = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
    colors = ['#3498db', '#e74c3c', '#2ecc71']
    
    # Plot each metric
    for idx, metric in enumerate(metrics):
        means = comparison_df[f'{metric}_mean'].values
        stds = comparison_df[f'{metric}_std'].values
        models = comparison_df['Model'].values
        
        # Bar plot with error bars
        bars = axes[idx].bar(range(len(models)), means, yerr=stds, 
                            capsize=5, color=colors, alpha=0.7, edgecolor='black')
        
        axes[idx].set_xticks(range(len(models)))
        axes[idx].set_xticklabels(models, rotation=45, ha='right', fontsize=9)
        axes[idx].set_ylabel('Score', fontsize=10)
        axes[idx].set_title(f'{metric.upper()} Comparison', fontsize=12, fontweight='bold')
        axes[idx].set_ylim([0, 1.0])
        axes[idx].grid(True, alpha=0.3, axis='y')
        
        # Add value labels on bars
        for i, (bar, mean, std) in enumerate(zip(bars, means, stds)):
            height = bar.get_height()
            axes[idx].text(bar.get_x() + bar.get_width()/2., height,
                          f'{mean:.3f}',
                          ha='center', va='bottom', fontsize=8, fontweight='bold')
    
    # Overall comparison (all metrics together)
    ax = axes[5]
    x = np.arange(len(metrics))
    width = 0.2
    
    for i, (model, color) in enumerate(zip(comparison_df['Model'], colors)):
        means = [comparison_df.loc[i, f'{m}_mean'] for m in metrics]
        ax.bar(x + i*width, means, width, label=model, color=color, alpha=0.7, edgecolor='black')
    
    ax.set_xlabel('Metric', fontsize=10)
    ax.set_ylabel('Score', fontsize=10)
    ax.set_title('Overall Model Comparison', fontsize=12, fontweight='bold')
    ax.set_xticks(x + width * 1.5)
    ax.set_xticklabels([m.upper() for m in metrics], fontsize=9)
    ax.set_ylim([0, 1.0])
    ax.legend(loc='lower right', fontsize=8)
    ax.grid(True, alpha=0.3, axis='y')
    
    plt.suptitle('10-Fold Cross-Validation Results - All Models', 
                fontsize=16, fontweight='bold', y=0.995)
    plt.tight_layout()
    plt.show()
    
    print("\n✓ Visualizations complete")

# Visualize comparison
# Visualize comparison
try:
    visualize_model_comparison(comparison_df)
except NameError:
    print("⚠️ Please run comparison first (Cell above)")

================================================================================
MODEL COMPARISON VISUALIZATIONS
================================================================================

✓ Visualizations complete

def create_detailed_comparison_table(comparison_df):
    """
    Create a detailed comparison table for reporting.
    """
    print("=" * 80)
    print("DETAILED COMPARISON TABLE FOR REPORTING")
    print("=" * 80)
    
    # Create formatted table
    report_df = pd.DataFrame()
    report_df['Model'] = comparison_df['Model']
    
    for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
        report_df[metric.capitalize()] = comparison_df.apply(
            lambda row: f"{row[f'{metric}_mean']:.4f} ± {row[f'{metric}_std']:.4f}",
            axis=1
        )
    
    print("\n📋 Copy this table for your report:\n")
    display(report_df)
    
    # Rank models
    print("\n" + "=" * 80)
    print("MODEL RANKINGS (by F1-Score)")
    print("=" * 80)
    
    ranked = comparison_df.sort_values('f1_mean', ascending=False)
    print("\n")
    for rank, (_, row) in enumerate(ranked.iterrows(), 1):
        medal = ['🥇', '🥈', '🥉', '  '][rank-1]
        print(f"{medal} {rank}. {row['Model']:<25} F1: {row['f1_mean']:.4f}")
    
    print("\n" + "=" * 80)
    
    return report_df

# Create detailed table
# Create detailed table
try:
    report_table = create_detailed_comparison_table(comparison_df)
except NameError:
    print("⚠️ Please run comparison first (Cell above)")

================================================================================
DETAILED COMPARISON TABLE FOR REPORTING
================================================================================

📋 Copy this table for your report:

================================================================================
MODEL RANKINGS (by F1-Score)
================================================================================


🥇 1. XGBoost                   F1: 1.0000
🥈 2. Random Forest             F1: 0.9999
🥉 3. Logistic Regression       F1: 0.9994

================================================================================

def analyze_overfitting(lr_results, rf_results, xgb_results):
    """
    Analyze potential overfitting by comparing train vs test performance.
    """
    print("=" * 80)
    print("OVERFITTING ANALYSIS")
    print("=" * 80)
    
    print("\nComparing training vs test performance (F1-Score):")
    print("Large gaps indicate potential overfitting\n")
    print("-" * 80)
    
    models_dict = {
        'Logistic Regression': lr_results,
        'Random Forest': rf_results,
        'XGBoost': xgb_results,
    }
    
    overfitting_data = []
    
    for model_name, results in models_dict.items():
        train_f1 = results['f1']['train_mean']
        test_f1 = results['f1']['test_mean']
        gap = train_f1 - test_f1
        
        overfitting_data.append({
            'Model': model_name,
            'Train_F1': train_f1,
            'Test_F1': test_f1,
            'Gap': gap,
            'Gap_Pct': (gap / train_f1) * 100 if train_f1 > 0 else 0
        })
        
        print(f"{model_name}:")
        print(f"  Train F1: {train_f1:.4f}")
        print(f"  Test F1:  {test_f1:.4f}")
        print(f"  Gap:      {gap:.4f} ({(gap/train_f1)*100:.1f}%)")
        
        if gap < 0.02:
            status = "✓ Excellent generalization"
        elif gap < 0.05:
            status = "✓ Good generalization"
        elif gap < 0.10:
            status = "⚠️ Slight overfitting"
        else:
            status = "❌ Significant overfitting"
        
        print(f"  Status:   {status}")
        print()
    
    # Visualize
    overfitting_df = pd.DataFrame(overfitting_data)
    
    fig, axes = plt.subplots(1, 2, figsize=(14, 5))
    
    # Plot 1: Train vs Test F1
    x = np.arange(len(overfitting_df))
    width = 0.35
    
    axes[0].bar(x - width/2, overfitting_df['Train_F1'], width, 
               label='Train F1', color='#3498db', alpha=0.7)
    axes[0].bar(x + width/2, overfitting_df['Test_F1'], width,
               label='Test F1', color='#e74c3c', alpha=0.7)
    
    axes[0].set_xlabel('Model')
    axes[0].set_ylabel('F1-Score')
    axes[0].set_title('Train vs Test F1-Score', fontweight='bold')
    axes[0].set_xticks(x)
    axes[0].set_xticklabels(overfitting_df['Model'], rotation=45, ha='right', fontsize=9)
    axes[0].legend()
    axes[0].grid(True, alpha=0.3, axis='y')
    
    # Plot 2: Overfitting gap
    colors = ['#2ecc71' if gap < 0.05 else '#f39c12' if gap < 0.10 else '#e74c3c' 
             for gap in overfitting_df['Gap']]
    
    axes[1].bar(range(len(overfitting_df)), overfitting_df['Gap'], 
               color=colors, alpha=0.7, edgecolor='black')
    axes[1].set_xlabel('Model')
    axes[1].set_ylabel('F1 Gap (Train - Test)')
    axes[1].set_title('Overfitting Gap Analysis', fontweight='bold')
    axes[1].set_xticks(range(len(overfitting_df)))
    axes[1].set_xticklabels(overfitting_df['Model'], rotation=45, ha='right', fontsize=9)
    axes[1].axhline(y=0.05, color='orange', linestyle='--', alpha=0.5, label='Acceptable gap')
    axes[1].axhline(y=0.10, color='red', linestyle='--', alpha=0.5, label='High gap')
    axes[1].legend()
    axes[1].grid(True, alpha=0.3, axis='y')
    
    plt.tight_layout()
    plt.show()
    
    print("=" * 80)
    
    return overfitting_df

# Analyze overfitting
try:
    overfitting_analysis = analyze_overfitting(lr_results, rf_results, xgb_results)
except NameError as e:
    print(f"⚠️ Please train all models first")
    print(f"   Missing: {e}")

================================================================================
OVERFITTING ANALYSIS
================================================================================

Comparing training vs test performance (F1-Score):
Large gaps indicate potential overfitting

--------------------------------------------------------------------------------
Logistic Regression:
  Train F1: 0.9995
  Test F1:  0.9994
  Gap:      0.0000 (0.0%)
  Status:   ✓ Excellent generalization

Random Forest:
  Train F1: 1.0000
  Test F1:  0.9999
  Gap:      0.0001 (0.0%)
  Status:   ✓ Excellent generalization

XGBoost:
  Train F1: 1.0000
  Test F1:  1.0000
  Gap:      0.0000 (0.0%)
  Status:   ✓ Excellent generalization

================================================================================

# Optional: Save results to CSV for reporting
# Uncomment to save:

# if 'report_table' in locals():
#     report_table.to_csv('model_comparison_results.csv', index=False)
#     print("✓ Results saved to 'model_comparison_results.csv'")

# if 'comparison_df' in locals():
#     comparison_df.to_csv('detailed_cv_results.csv', index=False)
#     print("✓ Detailed results saved to 'detailed_cv_results.csv'")

print("\n" + "=" * 80)
print("🎉 CLASSIFIER TRAINING AND EVALUATION COMPLETE!")
print("=" * 80)
print("\nYou have successfully:")
print("  ✓ Prepared features for machine learning")
print("  ✓ Trained 3 different classifiers")
print("  ✓ Evaluated with 10-fold cross-validation")
print("  ✓ Compared performance across multiple metrics")
print("  ✓ Analyzed overfitting")
print("\nReview the results above to determine the best model for your application!")
print("=" * 80)

================================================================================
🎉 CLASSIFIER TRAINING AND EVALUATION COMPLETE!
================================================================================

You have successfully:
  ✓ Prepared features for machine learning
  ✓ Trained 3 different classifiers
  ✓ Evaluated with 10-fold cross-validation
  ✓ Compared performance across multiple metrics
  ✓ Analyzed overfitting

Review the results above to determine the best model for your application!
================================================================================

# Define paths to your 7 CSV files
csv_files = [
    'Friday-WorkingHours-Afternoon-PortScan.pcap_ISCX.csv',  # <-- UPDATE THESE PATHS
    'Friday-WorkingHours-Morning.pcap_ISCX.csv',
    'Monday-WorkingHours.pcap_ISCX.csv',
    'Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv',
    'Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv',
    'Tuesday-WorkingHours.pcap_ISCX.csv',
    'Wednesday-workingHours.pcap_ISCX.csv'
]

print("=" * 80)
print("BATCH PROCESSING CONFIGURATION")
print("=" * 80)
print(f"\nNumber of files to process: {len(csv_files)}")
print(f"\nMethods:")
print(f"  • Unsupervised: DBSCAN clustering")
print(f"  • Supervised: XGBoost classifier (10-fold CV)")
print(f"\nFiles to process:")
for i, file in enumerate(csv_files, 1):
    print(f"  {i}. {file}")
print("\n" + "=" * 80)

================================================================================
BATCH PROCESSING CONFIGURATION
================================================================================

Number of files to process: 7

Methods:
  • Unsupervised: DBSCAN clustering
  • Supervised: XGBoost classifier (10-fold CV)

Files to process:
  1. Friday-WorkingHours-Afternoon-PortScan.pcap_ISCX.csv
  2. Friday-WorkingHours-Morning.pcap_ISCX.csv
  3. Monday-WorkingHours.pcap_ISCX.csv
  4. Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv
  5. Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv
  6. Tuesday-WorkingHours.pcap_ISCX.csv
  7. Wednesday-workingHours.pcap_ISCX.csv

================================================================================

def process_single_file(file_path, file_index, total_files):
    """
    Process a single CSV file through the entire pipeline.
    Returns results dictionary.
    """
    print("\n" + "#" * 80)
    print(f"# PROCESSING FILE {file_index}/{total_files}: {file_path}")
    print("#" * 80)
    
    results = {
        'file_path': file_path,
        'file_index': file_index,
        'success': False,
        'error': None
    }
    
    try:
        # =====================================================================
        # STEP 1: LOAD DATA
        # =====================================================================
        print("\n" + "=" * 80)
        print("STEP 1: LOADING DATA")
        print("=" * 80)
        
        df = load_data(file_path)
        
        if df is None:
            results['error'] = 'Failed to load data'
            return results
        
        results['n_samples'] = len(df)
        results['n_features'] = df.shape[1]
        
        # =====================================================================
        # STEP 2: CREATE BINARY LABELS
        # =====================================================================
        print("\n" + "=" * 80)
        print("STEP 2: CREATING BINARY LABELS")
        print("=" * 80)
        
        df_binary = create_binary_labels(df, label_column='Label', benign_value='BENIGN')
        
        if df_binary is None:
            results['error'] = 'Failed to create binary labels'
            return results
        
        # Store class distribution
        class_0_count = (df_binary['Binary_Label'] == 0).sum()
        class_1_count = (df_binary['Binary_Label'] == 1).sum()
        results['class_distribution'] = {
            'benign': class_0_count,
            'attack': class_1_count,
            'imbalance_ratio': class_0_count / class_1_count if class_1_count > 0 else float('inf')
        }
        
        # =====================================================================
        # STEP 3: UNSUPERVISED LEARNING - DBSCAN
        # =====================================================================
        print("\n" + "=" * 80)
        print("STEP 3: UNSUPERVISED LEARNING (DBSCAN)")
        print("=" * 80)
        
        # Preprocess for clustering
        X_scaled, true_labels, feature_names_clustering, scaler_clustering = preprocess_for_clustering(
            df_binary, 
            label_column='Binary_Label'
        )
        
        # Apply PCA
        X_pca, pca_model = apply_pca(X_scaled, n_components=0.95, max_components=20)
        
        # Run DBSCAN with reasonable default parameters
        # You can adjust eps and min_samples based on your data
        dbscan_labels, dbscan_model = apply_dbscan(
            X_pca, 
            eps=0.5,  # Adjust based on your data
            min_samples=10,  # Adjust based on your data
            sample_for_viz=5000
        )
        
        # Store DBSCAN results
        n_clusters = len(set(dbscan_labels)) - (1 if -1 in dbscan_labels else 0)
        n_noise = list(dbscan_labels).count(-1)
        
        results['dbscan'] = {
            'n_clusters': n_clusters,
            'n_noise': n_noise,
            'noise_percentage': (n_noise / len(dbscan_labels)) * 100
        }
        
        # =====================================================================
        # STEP 4: SUPERVISED LEARNING - XGBOOST
        # =====================================================================
        print("\n" + "=" * 80)
        print("STEP 4: SUPERVISED LEARNING (XGBOOST)")
        print("=" * 80)
        
        # Prepare features for ML
        X_ml, y_ml, feature_names_ml = prepare_features_for_ml(
            df_binary, 
            target_column='Binary_Label'
        )
        
        # Train XGBoost with 10-fold CV
        xgb_results, xgb_cv_results, xgb_model = train_xgboost(X_ml, y_ml, cv, scoring)
        
        # Store XGBoost results
        results['xgboost'] = {
            'accuracy': xgb_results['accuracy']['test_mean'],
            'precision': xgb_results['precision']['test_mean'],
            'recall': xgb_results['recall']['test_mean'],
            'f1': xgb_results['f1']['test_mean'],
            'roc_auc': xgb_results['roc_auc']['test_mean'],
            'accuracy_std': xgb_results['accuracy']['test_std'],
            'precision_std': xgb_results['precision']['test_std'],
            'recall_std': xgb_results['recall']['test_std'],
            'f1_std': xgb_results['f1']['test_std'],
            'roc_auc_std': xgb_results['roc_auc']['test_std']
        }
        
        results['success'] = True
        
        print("\n" + "=" * 80)
        print(f"✓ FILE {file_index}/{total_files} COMPLETED SUCCESSFULLY")
        print("=" * 80)
        
    except Exception as e:
        print(f"\n❌ ERROR processing {file_path}: {e}")
        import traceback
        traceback.print_exc()
        results['error'] = str(e)
    
    return results

print("✓ Processing function defined")

✓ Processing function defined

# Process all CSV files
print("=" * 80)
print("STARTING BATCH PROCESSING")
print("=" * 80)
print(f"\nProcessing {len(csv_files)} files...")
print(f"This may take 10-20 minutes depending on file sizes.\n")

import time
start_time = time.time()

# Store results for all files
all_results = []

# Process each file
for i, file_path in enumerate(csv_files, 1):
    result = process_single_file(file_path, i, len(csv_files))
    all_results.append(result)
    
    # Show progress
    if result['success']:
        print(f"\n✓ [{i}/{len(csv_files)}] {file_path} - SUCCESS")
    else:
        print(f"\n❌ [{i}/{len(csv_files)}] {file_path} - FAILED: {result['error']}")

elapsed = time.time() - start_time

print("\n" + "=" * 80)
print("BATCH PROCESSING COMPLETE")
print("=" * 80)
print(f"\nTotal time: {elapsed:.2f} seconds ({elapsed/60:.2f} minutes)")
print(f"Successfully processed: {sum(1 for r in all_results if r['success'])}/{len(csv_files)} files")
print("=" * 80)

================================================================================
STARTING BATCH PROCESSING
================================================================================

Processing 7 files...
This may take 10-20 minutes depending on file sizes.


################################################################################
# PROCESSING FILE 1/7: Friday-WorkingHours-Afternoon-PortScan.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================

✓ Data loaded successfully!
  Rows: 286,467
  Columns: 85
✓ Column names cleaned (whitespace stripped)

================================================================================
STEP 2: CREATING BINARY LABELS
================================================================================
================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  PortScan: 158,930 (55.48%)
  BENIGN: 127,537 (44.52%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0
  'PortScan' → 1

✓ Binary labels created in 'Binary_Label' column
================================================================================

================================================================================
STEP 3: UNSUPERVISED LEARNING (DBSCAN)
================================================================================
================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 286,467

2. Handling infinite values...
   Replaced 727 infinite values with NaN

3. Handling missing values...
   Imputed 742 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 27 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Flow IAT Std', 'Fwd IAT Total', 'Fwd IAT Mean', 'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Mean', 'Bwd IAT Max']

5. Features after correlation removal: 53

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 286,467 samples × 53 features
================================================================================
================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 53

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 91.35%

✓ PCA complete!
================================================================================
================================================================================
APPLYING DBSCAN CLUSTERING
================================================================================

Parameters:
  eps = 0.5
  min_samples = 10

Clustering 286,467 samples with 20 features...

✓ Clustering complete in 59.30 seconds

Results:
  Clusters found: 434
  Noise points: 21,763 (7.6%)
  Clustered points: 264,704 (92.4%)

Cluster size distribution:
  Cluster 0: 123 points (0.0%)
  Cluster 1: 1,288 points (0.4%)
  Cluster 2: 208 points (0.1%)
  Cluster 3: 25 points (0.0%)
  Cluster 4: 12 points (0.0%)
  Cluster 5: 60 points (0.0%)
  Cluster 6: 179 points (0.1%)
  Cluster 7: 51,076 points (17.8%)
  Cluster 8: 138 points (0.0%)
  Cluster 9: 3,518 points (1.2%)
  Cluster 10: 277 points (0.1%)
  Cluster 11: 174 points (0.1%)
  Cluster 12: 10 points (0.0%)
  Cluster 13: 21 points (0.0%)
  Cluster 14: 10 points (0.0%)
  Cluster 15: 38 points (0.0%)
  Cluster 16: 646 points (0.2%)
  Cluster 17: 38 points (0.0%)
  Cluster 18: 39 points (0.0%)
  Cluster 19: 58 points (0.0%)
  Cluster 20: 11 points (0.0%)
  Cluster 21: 22 points (0.0%)
  Cluster 22: 27 points (0.0%)
  Cluster 23: 73 points (0.0%)
  Cluster 24: 22 points (0.0%)
  Cluster 25: 472 points (0.2%)
  Cluster 26: 12 points (0.0%)
  Cluster 27: 15 points (0.0%)
  Cluster 28: 42 points (0.0%)
  Cluster 29: 10 points (0.0%)
  Cluster 30: 19 points (0.0%)
  Cluster 31: 404 points (0.1%)
  Cluster 32: 286 points (0.1%)
  Cluster 33: 50 points (0.0%)
  Cluster 34: 63 points (0.0%)
  Cluster 35: 13 points (0.0%)
  Cluster 36: 56 points (0.0%)
  Cluster 37: 24 points (0.0%)
  Cluster 38: 49 points (0.0%)
  Cluster 39: 5,948 points (2.1%)
  Cluster 40: 35 points (0.0%)
  Cluster 41: 38 points (0.0%)
  Cluster 42: 67 points (0.0%)
  Cluster 43: 16 points (0.0%)
  Cluster 44: 10 points (0.0%)
  Cluster 45: 2,521 points (0.9%)
  Cluster 46: 2,125 points (0.7%)
  Cluster 47: 2,762 points (1.0%)
  Cluster 48: 16 points (0.0%)
  Cluster 49: 50 points (0.0%)
  Cluster 50: 54 points (0.0%)
  Cluster 51: 20 points (0.0%)
  Cluster 52: 23 points (0.0%)
  Cluster 53: 58 points (0.0%)
  Cluster 54: 13 points (0.0%)
  Cluster 55: 14 points (0.0%)
  Cluster 56: 12 points (0.0%)
  Cluster 57: 20 points (0.0%)
  Cluster 58: 773 points (0.3%)
  Cluster 59: 2,771 points (1.0%)
  Cluster 60: 68 points (0.0%)
  Cluster 61: 203 points (0.1%)
  Cluster 62: 354 points (0.1%)
  Cluster 63: 53 points (0.0%)
  Cluster 64: 111 points (0.0%)
  Cluster 65: 183 points (0.1%)
  Cluster 66: 70 points (0.0%)
  Cluster 67: 366 points (0.1%)
  Cluster 68: 13 points (0.0%)
  Cluster 69: 202 points (0.1%)
  Cluster 70: 101 points (0.0%)
  Cluster 71: 81 points (0.0%)
  Cluster 72: 1,808 points (0.6%)
  Cluster 73: 492 points (0.2%)
  Cluster 74: 246 points (0.1%)
  Cluster 75: 255 points (0.1%)
  Cluster 76: 469 points (0.2%)
  Cluster 77: 1,071 points (0.4%)
  Cluster 78: 134 points (0.0%)
  Cluster 79: 797 points (0.3%)
  Cluster 80: 217 points (0.1%)
  Cluster 81: 258 points (0.1%)
  Cluster 82: 92 points (0.0%)
  Cluster 83: 105 points (0.0%)
  Cluster 84: 77 points (0.0%)
  Cluster 85: 751 points (0.3%)
  Cluster 86: 1,120 points (0.4%)
  Cluster 87: 77,001 points (26.9%)
  Cluster 88: 222 points (0.1%)
  Cluster 89: 115 points (0.0%)
  Cluster 90: 90 points (0.0%)
  Cluster 91: 153 points (0.1%)
  Cluster 92: 124 points (0.0%)
  Cluster 93: 615 points (0.2%)
  Cluster 94: 30 points (0.0%)
  Cluster 95: 554 points (0.2%)
  Cluster 96: 160 points (0.1%)
  Cluster 97: 172 points (0.1%)
  Cluster 98: 29 points (0.0%)
  Cluster 99: 30 points (0.0%)
  Cluster 100: 122 points (0.0%)
  Cluster 101: 75 points (0.0%)
  Cluster 102: 24 points (0.0%)
  Cluster 103: 15 points (0.0%)
  Cluster 104: 393 points (0.1%)
  Cluster 105: 33 points (0.0%)
  Cluster 106: 44 points (0.0%)
  Cluster 107: 704 points (0.2%)
  Cluster 108: 25 points (0.0%)
  Cluster 109: 267 points (0.1%)
  Cluster 110: 94 points (0.0%)
  Cluster 111: 11 points (0.0%)
  Cluster 112: 57 points (0.0%)
  Cluster 113: 114 points (0.0%)
  Cluster 114: 13 points (0.0%)
  Cluster 115: 13 points (0.0%)
  Cluster 116: 20 points (0.0%)
  Cluster 117: 15 points (0.0%)
  Cluster 118: 79 points (0.0%)
  Cluster 119: 398 points (0.1%)
  Cluster 120: 20 points (0.0%)
  Cluster 121: 40 points (0.0%)
  Cluster 122: 50 points (0.0%)
  Cluster 123: 393 points (0.1%)
  Cluster 124: 82 points (0.0%)
  Cluster 125: 120 points (0.0%)
  Cluster 126: 390 points (0.1%)
  Cluster 127: 111 points (0.0%)
  Cluster 128: 527 points (0.2%)
  Cluster 129: 67 points (0.0%)
  Cluster 130: 54 points (0.0%)
  Cluster 131: 236 points (0.1%)
  Cluster 132: 42 points (0.0%)
  Cluster 133: 13 points (0.0%)
  Cluster 134: 155 points (0.1%)
  Cluster 135: 271 points (0.1%)
  Cluster 136: 42 points (0.0%)
  Cluster 137: 39 points (0.0%)
  Cluster 138: 126 points (0.0%)
  Cluster 139: 40 points (0.0%)
  Cluster 140: 297 points (0.1%)
  Cluster 141: 12 points (0.0%)
  Cluster 142: 90 points (0.0%)
  Cluster 143: 98 points (0.0%)
  Cluster 144: 96 points (0.0%)
  Cluster 145: 41 points (0.0%)
  Cluster 146: 25 points (0.0%)
  Cluster 147: 138 points (0.0%)
  Cluster 148: 132 points (0.0%)
  Cluster 149: 34 points (0.0%)
  Cluster 150: 11 points (0.0%)
  Cluster 151: 10 points (0.0%)
  Cluster 152: 54 points (0.0%)
  Cluster 153: 86 points (0.0%)
  Cluster 154: 18 points (0.0%)
  Cluster 155: 71 points (0.0%)
  Cluster 156: 15 points (0.0%)
  Cluster 157: 182 points (0.1%)
  Cluster 158: 807 points (0.3%)
  Cluster 159: 14 points (0.0%)
  Cluster 160: 19 points (0.0%)
  Cluster 161: 31 points (0.0%)
  Cluster 162: 61 points (0.0%)
  Cluster 163: 103 points (0.0%)
  Cluster 164: 69 points (0.0%)
  Cluster 165: 85 points (0.0%)
  Cluster 166: 15 points (0.0%)
  Cluster 167: 56 points (0.0%)
  Cluster 168: 47 points (0.0%)
  Cluster 169: 32 points (0.0%)
  Cluster 170: 10 points (0.0%)
  Cluster 171: 149 points (0.1%)
  Cluster 172: 62 points (0.0%)
  Cluster 173: 18 points (0.0%)
  Cluster 174: 58 points (0.0%)
  Cluster 175: 112 points (0.0%)
  Cluster 176: 39 points (0.0%)
  Cluster 177: 165 points (0.1%)
  Cluster 178: 48 points (0.0%)
  Cluster 179: 44 points (0.0%)
  Cluster 180: 13 points (0.0%)
  Cluster 181: 155 points (0.1%)
  Cluster 182: 11 points (0.0%)
  Cluster 183: 13 points (0.0%)
  Cluster 184: 91 points (0.0%)
  Cluster 185: 44 points (0.0%)
  Cluster 186: 26 points (0.0%)
  Cluster 187: 19 points (0.0%)
  Cluster 188: 116 points (0.0%)
  Cluster 189: 375 points (0.1%)
  Cluster 190: 119 points (0.0%)
  Cluster 191: 182 points (0.1%)
  Cluster 192: 20 points (0.0%)
  Cluster 193: 312 points (0.1%)
  Cluster 194: 46 points (0.0%)
  Cluster 195: 49 points (0.0%)
  Cluster 196: 57 points (0.0%)
  Cluster 197: 16 points (0.0%)
  Cluster 198: 38 points (0.0%)
  Cluster 199: 20 points (0.0%)
  Cluster 200: 11 points (0.0%)
  Cluster 201: 76 points (0.0%)
  Cluster 202: 8 points (0.0%)
  Cluster 203: 110 points (0.0%)
  Cluster 204: 33 points (0.0%)
  Cluster 205: 58 points (0.0%)
  Cluster 206: 13 points (0.0%)
  Cluster 207: 32 points (0.0%)
  Cluster 208: 45 points (0.0%)
  Cluster 209: 22 points (0.0%)
  Cluster 210: 39 points (0.0%)
  Cluster 211: 79 points (0.0%)
  Cluster 212: 96 points (0.0%)
  Cluster 213: 15 points (0.0%)
  Cluster 214: 25 points (0.0%)
  Cluster 215: 17 points (0.0%)
  Cluster 216: 45 points (0.0%)
  Cluster 217: 37 points (0.0%)
  Cluster 218: 21 points (0.0%)
  Cluster 219: 36 points (0.0%)
  Cluster 220: 13 points (0.0%)
  Cluster 221: 22 points (0.0%)
  Cluster 222: 22 points (0.0%)
  Cluster 223: 19 points (0.0%)
  Cluster 224: 125 points (0.0%)
  Cluster 225: 35 points (0.0%)
  Cluster 226: 67 points (0.0%)
  Cluster 227: 69 points (0.0%)
  Cluster 228: 22 points (0.0%)
  Cluster 229: 18 points (0.0%)
  Cluster 230: 11 points (0.0%)
  Cluster 231: 32 points (0.0%)
  Cluster 232: 30 points (0.0%)
  Cluster 233: 91 points (0.0%)
  Cluster 234: 30 points (0.0%)
  Cluster 235: 19 points (0.0%)
  Cluster 236: 10 points (0.0%)
  Cluster 237: 36 points (0.0%)
  Cluster 238: 58 points (0.0%)
  Cluster 239: 46 points (0.0%)
  Cluster 240: 13 points (0.0%)
  Cluster 241: 64 points (0.0%)
  Cluster 242: 73 points (0.0%)
  Cluster 243: 44 points (0.0%)
  Cluster 244: 26 points (0.0%)
  Cluster 245: 74 points (0.0%)
  Cluster 246: 33 points (0.0%)
  Cluster 247: 104 points (0.0%)
  Cluster 248: 19 points (0.0%)
  Cluster 249: 11 points (0.0%)
  Cluster 250: 67 points (0.0%)
  Cluster 251: 11 points (0.0%)
  Cluster 252: 75 points (0.0%)
  Cluster 253: 109 points (0.0%)
  Cluster 254: 17 points (0.0%)
  Cluster 255: 37 points (0.0%)
  Cluster 256: 29 points (0.0%)
  Cluster 257: 19 points (0.0%)
  Cluster 258: 10 points (0.0%)
  Cluster 259: 26 points (0.0%)
  Cluster 260: 17 points (0.0%)
  Cluster 261: 17 points (0.0%)
  Cluster 262: 13 points (0.0%)
  Cluster 263: 11 points (0.0%)
  Cluster 264: 13 points (0.0%)
  Cluster 265: 23 points (0.0%)
  Cluster 266: 177 points (0.1%)
  Cluster 267: 45 points (0.0%)
  Cluster 268: 34 points (0.0%)
  Cluster 269: 17 points (0.0%)
  Cluster 270: 10 points (0.0%)
  Cluster 271: 60 points (0.0%)
  Cluster 272: 10 points (0.0%)
  Cluster 273: 23 points (0.0%)
  Cluster 274: 30 points (0.0%)
  Cluster 275: 20 points (0.0%)
  Cluster 276: 60 points (0.0%)
  Cluster 277: 17 points (0.0%)
  Cluster 278: 27 points (0.0%)
  Cluster 279: 74 points (0.0%)
  Cluster 280: 26 points (0.0%)
  Cluster 281: 29 points (0.0%)
  Cluster 282: 17 points (0.0%)
  Cluster 283: 17 points (0.0%)
  Cluster 284: 78,489 points (27.4%)
  Cluster 285: 14 points (0.0%)
  Cluster 286: 40 points (0.0%)
  Cluster 287: 113 points (0.0%)
  Cluster 288: 11 points (0.0%)
  Cluster 289: 24 points (0.0%)
  Cluster 290: 13 points (0.0%)
  Cluster 291: 10 points (0.0%)
  Cluster 292: 19 points (0.0%)
  Cluster 293: 10 points (0.0%)
  Cluster 294: 73 points (0.0%)
  Cluster 295: 17 points (0.0%)
  Cluster 296: 14 points (0.0%)
  Cluster 297: 13 points (0.0%)
  Cluster 298: 28 points (0.0%)
  Cluster 299: 27 points (0.0%)
  Cluster 300: 48 points (0.0%)
  Cluster 301: 10 points (0.0%)
  Cluster 302: 42 points (0.0%)
  Cluster 303: 17 points (0.0%)
  Cluster 304: 40 points (0.0%)
  Cluster 305: 27 points (0.0%)
  Cluster 306: 18 points (0.0%)
  Cluster 307: 23 points (0.0%)
  Cluster 308: 55 points (0.0%)
  Cluster 309: 12 points (0.0%)
  Cluster 310: 14 points (0.0%)
  Cluster 311: 258 points (0.1%)
  Cluster 312: 45 points (0.0%)
  Cluster 313: 74 points (0.0%)
  Cluster 314: 15 points (0.0%)
  Cluster 315: 28 points (0.0%)
  Cluster 316: 19 points (0.0%)
  Cluster 317: 11 points (0.0%)
  Cluster 318: 16 points (0.0%)
  Cluster 319: 13 points (0.0%)
  Cluster 320: 10 points (0.0%)
  Cluster 321: 15 points (0.0%)
  Cluster 322: 47 points (0.0%)
  Cluster 323: 17 points (0.0%)
  Cluster 324: 22 points (0.0%)
  Cluster 325: 20 points (0.0%)
  Cluster 326: 493 points (0.2%)
  Cluster 327: 10 points (0.0%)
  Cluster 328: 26 points (0.0%)
  Cluster 329: 18 points (0.0%)
  Cluster 330: 110 points (0.0%)
  Cluster 331: 18 points (0.0%)
  Cluster 332: 42 points (0.0%)
  Cluster 333: 16 points (0.0%)
  Cluster 334: 17 points (0.0%)
  Cluster 335: 19 points (0.0%)
  Cluster 336: 16 points (0.0%)
  Cluster 337: 36 points (0.0%)
  Cluster 338: 14 points (0.0%)
  Cluster 339: 10 points (0.0%)
  Cluster 340: 17 points (0.0%)
  Cluster 341: 27 points (0.0%)
  Cluster 342: 17 points (0.0%)
  Cluster 343: 15 points (0.0%)
  Cluster 344: 11 points (0.0%)
  Cluster 345: 11 points (0.0%)
  Cluster 346: 23 points (0.0%)
  Cluster 347: 14 points (0.0%)
  Cluster 348: 484 points (0.2%)
  Cluster 349: 45 points (0.0%)
  Cluster 350: 20 points (0.0%)
  Cluster 351: 12 points (0.0%)
  Cluster 352: 15 points (0.0%)
  Cluster 353: 19 points (0.0%)
  Cluster 354: 16 points (0.0%)
  Cluster 355: 14 points (0.0%)
  Cluster 356: 10 points (0.0%)
  Cluster 357: 12 points (0.0%)
  Cluster 358: 11 points (0.0%)
  Cluster 359: 17 points (0.0%)
  Cluster 360: 62 points (0.0%)
  Cluster 361: 22 points (0.0%)
  Cluster 362: 13 points (0.0%)
  Cluster 363: 122 points (0.0%)
  Cluster 364: 12 points (0.0%)
  Cluster 365: 15 points (0.0%)
  Cluster 366: 11 points (0.0%)
  Cluster 367: 26 points (0.0%)
  Cluster 368: 14 points (0.0%)
  Cluster 369: 10 points (0.0%)
  Cluster 370: 12 points (0.0%)
  Cluster 371: 37 points (0.0%)
  Cluster 372: 11 points (0.0%)
  Cluster 373: 10 points (0.0%)
  Cluster 374: 10 points (0.0%)
  Cluster 375: 7 points (0.0%)
  Cluster 376: 11 points (0.0%)
  Cluster 377: 9 points (0.0%)
  Cluster 378: 20 points (0.0%)
  Cluster 379: 10 points (0.0%)
  Cluster 380: 10 points (0.0%)
  Cluster 381: 15 points (0.0%)
  Cluster 382: 22 points (0.0%)
  Cluster 383: 155 points (0.1%)
  Cluster 384: 217 points (0.1%)
  Cluster 385: 63 points (0.0%)
  Cluster 386: 232 points (0.1%)
  Cluster 387: 167 points (0.1%)
  Cluster 388: 178 points (0.1%)
  Cluster 389: 35 points (0.0%)
  Cluster 390: 17 points (0.0%)
  Cluster 391: 16 points (0.0%)
  Cluster 392: 11 points (0.0%)
  Cluster 393: 10 points (0.0%)
  Cluster 394: 10 points (0.0%)
  Cluster 395: 10 points (0.0%)
  Cluster 396: 189 points (0.1%)
  Cluster 397: 67 points (0.0%)
  Cluster 398: 20 points (0.0%)
  Cluster 399: 233 points (0.1%)
  Cluster 400: 55 points (0.0%)
  Cluster 401: 42 points (0.0%)
  Cluster 402: 116 points (0.0%)
  Cluster 403: 181 points (0.1%)
  Cluster 404: 179 points (0.1%)
  Cluster 405: 177 points (0.1%)
  Cluster 406: 123 points (0.0%)
  Cluster 407: 12 points (0.0%)
  Cluster 408: 10 points (0.0%)
  Cluster 409: 29 points (0.0%)
  Cluster 410: 70 points (0.0%)
  Cluster 411: 65 points (0.0%)
  Cluster 412: 7 points (0.0%)
  Cluster 413: 10 points (0.0%)
  Cluster 414: 10 points (0.0%)
  Cluster 415: 14 points (0.0%)
  Cluster 416: 10 points (0.0%)
  Cluster 417: 10 points (0.0%)
  Cluster 418: 10 points (0.0%)
  Cluster 419: 10 points (0.0%)
  Cluster 420: 19 points (0.0%)
  Cluster 421: 19 points (0.0%)
  Cluster 422: 37 points (0.0%)
  Cluster 423: 19 points (0.0%)
  Cluster 424: 15 points (0.0%)
  Cluster 425: 10 points (0.0%)
  Cluster 426: 15 points (0.0%)
  Cluster 427: 11 points (0.0%)
  Cluster 428: 15 points (0.0%)
  Cluster 429: 10 points (0.0%)
  Cluster 430: 23 points (0.0%)
  Cluster 431: 11 points (0.0%)
  Cluster 432: 38 points (0.0%)
  Cluster 433: 18 points (0.0%)

Generating visualization...

================================================================================

================================================================================
STEP 4: SUPERVISED LEARNING (XGBOOST)
================================================================================
================================================================================
FEATURE PREPARATION FOR MACHINE LEARNING
================================================================================

1. Removing 8 non-feature columns:
   - Flow ID
   - Source IP
   - Destination IP
   - Timestamp
   - Label
   - Binary_Label
   - Source Port
   - Destination Port

2. Ensuring all features are numeric...
   ✓ All features are numeric

3. Handling infinite values...
   Found 727 infinite values
   ✓ Replaced with NaN

4. Handling missing values...
   Found 742 missing values
   Imputing with column medians...
   ✓ Missing values imputed

================================================================================
FINAL DATASET FOR MODELING
================================================================================

Samples: 286,467
Features: 78

Class distribution:
  Class 0 (BENIGN): 127,537 (44.52%)
  Class 1 (ATTACK): 158,930 (55.48%)

Feature types:
  Port features: 0
  Other features: 78

Sample feature names:
  1. Protocol
  2. Flow Duration
  3. Total Fwd Packets
  4. Total Backward Packets
  5. Total Length of Fwd Packets
  6. Total Length of Bwd Packets
  7. Fwd Packet Length Max
  8. Fwd Packet Length Min
  9. Fwd Packet Length Mean
  10. Fwd Packet Length Std
  ... and 68 more

================================================================================
================================================================================
MODEL 3: XGBOOST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Calculated scale_pos_weight: 0.80
  (Ratio of negative to positive samples)

Creating XGBoost model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 5.36 seconds

================================================================================
XGBOOST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

PRECISION:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

RECALL:
  Test:  0.9999 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

F1:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

ROC_AUC:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

================================================================================

================================================================================
✓ FILE 1/7 COMPLETED SUCCESSFULLY
================================================================================

✓ [1/7] Friday-WorkingHours-Afternoon-PortScan.pcap_ISCX.csv - SUCCESS

################################################################################
# PROCESSING FILE 2/7: Friday-WorkingHours-Morning.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================
✓ Data loaded successfully!
  Rows: 191,033
  Columns: 85
✓ Column names cleaned (whitespace stripped)

================================================================================
STEP 2: CREATING BINARY LABELS
================================================================================
================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  BENIGN: 189,067 (98.97%)
  Bot: 1,966 (1.03%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0
  'Bot' → 1

✓ Binary labels created in 'Binary_Label' column
================================================================================

================================================================================
STEP 3: UNSUPERVISED LEARNING (DBSCAN)
================================================================================
================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 191,033

2. Handling infinite values...
   Replaced 216 infinite values with NaN

3. Handling missing values...
   Imputed 244 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 24 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Fwd IAT Total', 'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Min', 'Fwd Header Length', 'Bwd Header Length', 'Fwd Packets/s']

5. Features after correlation removal: 56

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 191,033 samples × 56 features
================================================================================
================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 56

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 91.48%

✓ PCA complete!
================================================================================
================================================================================
APPLYING DBSCAN CLUSTERING
================================================================================

Parameters:
  eps = 0.5
  min_samples = 10

Clustering 191,033 samples with 20 features...

✓ Clustering complete in 9.83 seconds

Results:
  Clusters found: 452
  Noise points: 21,852 (11.4%)
  Clustered points: 169,181 (88.6%)

Cluster size distribution:
  Cluster 0: 12 points (0.0%)
  Cluster 1: 12 points (0.0%)
  Cluster 2: 759 points (0.4%)
  Cluster 3: 492 points (0.3%)
  Cluster 4: 1,080 points (0.6%)
  Cluster 5: 29 points (0.0%)
  Cluster 6: 51 points (0.0%)
  Cluster 7: 41,379 points (21.7%)
  Cluster 8: 41,640 points (21.8%)
  Cluster 9: 143 points (0.1%)
  Cluster 10: 22 points (0.0%)
  Cluster 11: 27 points (0.0%)
  Cluster 12: 101 points (0.1%)
  Cluster 13: 1,082 points (0.6%)
  Cluster 14: 2,944 points (1.5%)
  Cluster 15: 34 points (0.0%)
  Cluster 16: 38 points (0.0%)
  Cluster 17: 33 points (0.0%)
  Cluster 18: 100 points (0.1%)
  Cluster 19: 3,326 points (1.7%)
  Cluster 20: 86 points (0.0%)
  Cluster 21: 78 points (0.0%)
  Cluster 22: 26 points (0.0%)
  Cluster 23: 48 points (0.0%)
  Cluster 24: 707 points (0.4%)
  Cluster 25: 19 points (0.0%)
  Cluster 26: 20 points (0.0%)
  Cluster 27: 52 points (0.0%)
  Cluster 28: 6,724 points (3.5%)
  Cluster 29: 87 points (0.0%)
  Cluster 30: 54 points (0.0%)
  Cluster 31: 56 points (0.0%)
  Cluster 32: 23 points (0.0%)
  Cluster 33: 58 points (0.0%)
  Cluster 34: 17 points (0.0%)
  Cluster 35: 88 points (0.0%)
  Cluster 36: 788 points (0.4%)
  Cluster 37: 35 points (0.0%)
  Cluster 38: 236 points (0.1%)
  Cluster 39: 11 points (0.0%)
  Cluster 40: 3,792 points (2.0%)
  Cluster 41: 38 points (0.0%)
  Cluster 42: 76 points (0.0%)
  Cluster 43: 11 points (0.0%)
  Cluster 44: 11 points (0.0%)
  Cluster 45: 63 points (0.0%)
  Cluster 46: 54 points (0.0%)
  Cluster 47: 17 points (0.0%)
  Cluster 48: 739 points (0.4%)
  Cluster 49: 105 points (0.1%)
  Cluster 50: 75 points (0.0%)
  Cluster 51: 15 points (0.0%)
  Cluster 52: 25 points (0.0%)
  Cluster 53: 62 points (0.0%)
  Cluster 54: 56 points (0.0%)
  Cluster 55: 90 points (0.0%)
  Cluster 56: 20 points (0.0%)
  Cluster 57: 116 points (0.1%)
  Cluster 58: 15 points (0.0%)
  Cluster 59: 21 points (0.0%)
  Cluster 60: 837 points (0.4%)
  Cluster 61: 725 points (0.4%)
  Cluster 62: 11 points (0.0%)
  Cluster 63: 3,545 points (1.9%)
  Cluster 64: 385 points (0.2%)
  Cluster 65: 1,604 points (0.8%)
  Cluster 66: 15 points (0.0%)
  Cluster 67: 237 points (0.1%)
  Cluster 68: 89 points (0.0%)
  Cluster 69: 610 points (0.3%)
  Cluster 70: 245 points (0.1%)
  Cluster 71: 35 points (0.0%)
  Cluster 72: 19 points (0.0%)
  Cluster 73: 933 points (0.5%)
  Cluster 74: 15 points (0.0%)
  Cluster 75: 14 points (0.0%)
  Cluster 76: 118 points (0.1%)
  Cluster 77: 169 points (0.1%)
  Cluster 78: 23 points (0.0%)
  Cluster 79: 155 points (0.1%)
  Cluster 80: 140 points (0.1%)
  Cluster 81: 1,108 points (0.6%)
  Cluster 82: 88 points (0.0%)
  Cluster 83: 541 points (0.3%)
  Cluster 84: 326 points (0.2%)
  Cluster 85: 263 points (0.1%)
  Cluster 86: 512 points (0.3%)
  Cluster 87: 22 points (0.0%)
  Cluster 88: 513 points (0.3%)
  Cluster 89: 108 points (0.1%)
  Cluster 90: 26 points (0.0%)
  Cluster 91: 503 points (0.3%)
  Cluster 92: 508 points (0.3%)
  Cluster 93: 792 points (0.4%)
  Cluster 94: 74 points (0.0%)
  Cluster 95: 40 points (0.0%)
  Cluster 96: 87 points (0.0%)
  Cluster 97: 23 points (0.0%)
  Cluster 98: 11 points (0.0%)
  Cluster 99: 177 points (0.1%)
  Cluster 100: 992 points (0.5%)
  Cluster 101: 18 points (0.0%)
  Cluster 102: 26 points (0.0%)
  Cluster 103: 897 points (0.5%)
  Cluster 104: 945 points (0.5%)
  Cluster 105: 326 points (0.2%)
  Cluster 106: 142 points (0.1%)
  Cluster 107: 601 points (0.3%)
  Cluster 108: 450 points (0.2%)
  Cluster 109: 92 points (0.0%)
  Cluster 110: 176 points (0.1%)
  Cluster 111: 172 points (0.1%)
  Cluster 112: 799 points (0.4%)
  Cluster 113: 106 points (0.1%)
  Cluster 114: 30 points (0.0%)
  Cluster 115: 152 points (0.1%)
  Cluster 116: 14 points (0.0%)
  Cluster 117: 114 points (0.1%)
  Cluster 118: 150 points (0.1%)
  Cluster 119: 26 points (0.0%)
  Cluster 120: 30 points (0.0%)
  Cluster 121: 2,088 points (1.1%)
  Cluster 122: 1,457 points (0.8%)
  Cluster 123: 152 points (0.1%)
  Cluster 124: 3,962 points (2.1%)
  Cluster 125: 2,395 points (1.3%)
  Cluster 126: 1,724 points (0.9%)
  Cluster 127: 852 points (0.4%)
  Cluster 128: 464 points (0.2%)
  Cluster 129: 85 points (0.0%)
  Cluster 130: 2,313 points (1.2%)
  Cluster 131: 121 points (0.1%)
  Cluster 132: 173 points (0.1%)
  Cluster 133: 1,444 points (0.8%)
  Cluster 134: 184 points (0.1%)
  Cluster 135: 124 points (0.1%)
  Cluster 136: 14 points (0.0%)
  Cluster 137: 170 points (0.1%)
  Cluster 138: 164 points (0.1%)
  Cluster 139: 212 points (0.1%)
  Cluster 140: 40 points (0.0%)
  Cluster 141: 742 points (0.4%)
  Cluster 142: 11 points (0.0%)
  Cluster 143: 23 points (0.0%)
  Cluster 144: 167 points (0.1%)
  Cluster 145: 67 points (0.0%)
  Cluster 146: 4,850 points (2.5%)
  Cluster 147: 11 points (0.0%)
  Cluster 148: 232 points (0.1%)
  Cluster 149: 66 points (0.0%)
  Cluster 150: 55 points (0.0%)
  Cluster 151: 43 points (0.0%)
  Cluster 152: 104 points (0.1%)
  Cluster 153: 110 points (0.1%)
  Cluster 154: 46 points (0.0%)
  Cluster 155: 17 points (0.0%)
  Cluster 156: 17 points (0.0%)
  Cluster 157: 11 points (0.0%)
  Cluster 158: 65 points (0.0%)
  Cluster 159: 105 points (0.1%)
  Cluster 160: 17 points (0.0%)
  Cluster 161: 870 points (0.5%)
  Cluster 162: 30 points (0.0%)
  Cluster 163: 37 points (0.0%)
  Cluster 164: 15 points (0.0%)
  Cluster 165: 14 points (0.0%)
  Cluster 166: 772 points (0.4%)
  Cluster 167: 243 points (0.1%)
  Cluster 168: 84 points (0.0%)
  Cluster 169: 22 points (0.0%)
  Cluster 170: 113 points (0.1%)
  Cluster 171: 108 points (0.1%)
  Cluster 172: 113 points (0.1%)
  Cluster 173: 26 points (0.0%)
  Cluster 174: 27 points (0.0%)
  Cluster 175: 147 points (0.1%)
  Cluster 176: 77 points (0.0%)
  Cluster 177: 35 points (0.0%)
  Cluster 178: 64 points (0.0%)
  Cluster 179: 161 points (0.1%)
  Cluster 180: 14 points (0.0%)
  Cluster 181: 16 points (0.0%)
  Cluster 182: 10 points (0.0%)
  Cluster 183: 17 points (0.0%)
  Cluster 184: 287 points (0.2%)
  Cluster 185: 12 points (0.0%)
  Cluster 186: 308 points (0.2%)
  Cluster 187: 428 points (0.2%)
  Cluster 188: 383 points (0.2%)
  Cluster 189: 64 points (0.0%)
  Cluster 190: 104 points (0.1%)
  Cluster 191: 179 points (0.1%)
  Cluster 192: 145 points (0.1%)
  Cluster 193: 50 points (0.0%)
  Cluster 194: 99 points (0.1%)
  Cluster 195: 157 points (0.1%)
  Cluster 196: 15 points (0.0%)
  Cluster 197: 41 points (0.0%)
  Cluster 198: 81 points (0.0%)
  Cluster 199: 106 points (0.1%)
  Cluster 200: 30 points (0.0%)
  Cluster 201: 31 points (0.0%)
  Cluster 202: 17 points (0.0%)
  Cluster 203: 16 points (0.0%)
  Cluster 204: 22 points (0.0%)
  Cluster 205: 15 points (0.0%)
  Cluster 206: 38 points (0.0%)
  Cluster 207: 52 points (0.0%)
  Cluster 208: 37 points (0.0%)
  Cluster 209: 18 points (0.0%)
  Cluster 210: 58 points (0.0%)
  Cluster 211: 59 points (0.0%)
  Cluster 212: 69 points (0.0%)
  Cluster 213: 74 points (0.0%)
  Cluster 214: 57 points (0.0%)
  Cluster 215: 489 points (0.3%)
  Cluster 216: 33 points (0.0%)
  Cluster 217: 99 points (0.1%)
  Cluster 218: 19 points (0.0%)
  Cluster 219: 13 points (0.0%)
  Cluster 220: 27 points (0.0%)
  Cluster 221: 14 points (0.0%)
  Cluster 222: 14 points (0.0%)
  Cluster 223: 32 points (0.0%)
  Cluster 224: 103 points (0.1%)
  Cluster 225: 10 points (0.0%)
  Cluster 226: 143 points (0.1%)
  Cluster 227: 19 points (0.0%)
  Cluster 228: 30 points (0.0%)
  Cluster 229: 13 points (0.0%)
  Cluster 230: 46 points (0.0%)
  Cluster 231: 10 points (0.0%)
  Cluster 232: 54 points (0.0%)
  Cluster 233: 36 points (0.0%)
  Cluster 234: 14 points (0.0%)
  Cluster 235: 15 points (0.0%)
  Cluster 236: 53 points (0.0%)
  Cluster 237: 41 points (0.0%)
  Cluster 238: 20 points (0.0%)
  Cluster 239: 70 points (0.0%)
  Cluster 240: 71 points (0.0%)
  Cluster 241: 13 points (0.0%)
  Cluster 242: 15 points (0.0%)
  Cluster 243: 11 points (0.0%)
  Cluster 244: 24 points (0.0%)
  Cluster 245: 18 points (0.0%)
  Cluster 246: 19 points (0.0%)
  Cluster 247: 13 points (0.0%)
  Cluster 248: 10 points (0.0%)
  Cluster 249: 170 points (0.1%)
  Cluster 250: 58 points (0.0%)
  Cluster 251: 21 points (0.0%)
  Cluster 252: 11 points (0.0%)
  Cluster 253: 13 points (0.0%)
  Cluster 254: 33 points (0.0%)
  Cluster 255: 19 points (0.0%)
  Cluster 256: 66 points (0.0%)
  Cluster 257: 19 points (0.0%)
  Cluster 258: 18 points (0.0%)
  Cluster 259: 66 points (0.0%)
  Cluster 260: 31 points (0.0%)
  Cluster 261: 51 points (0.0%)
  Cluster 262: 55 points (0.0%)
  Cluster 263: 130 points (0.1%)
  Cluster 264: 13 points (0.0%)
  Cluster 265: 188 points (0.1%)
  Cluster 266: 40 points (0.0%)
  Cluster 267: 22 points (0.0%)
  Cluster 268: 50 points (0.0%)
  Cluster 269: 14 points (0.0%)
  Cluster 270: 65 points (0.0%)
  Cluster 271: 13 points (0.0%)
  Cluster 272: 16 points (0.0%)
  Cluster 273: 17 points (0.0%)
  Cluster 274: 98 points (0.1%)
  Cluster 275: 12 points (0.0%)
  Cluster 276: 235 points (0.1%)
  Cluster 277: 12 points (0.0%)
  Cluster 278: 41 points (0.0%)
  Cluster 279: 15 points (0.0%)
  Cluster 280: 11 points (0.0%)
  Cluster 281: 10 points (0.0%)
  Cluster 282: 34 points (0.0%)
  Cluster 283: 10 points (0.0%)
  Cluster 284: 24 points (0.0%)
  Cluster 285: 24 points (0.0%)
  Cluster 286: 11 points (0.0%)
  Cluster 287: 24 points (0.0%)
  Cluster 288: 240 points (0.1%)
  Cluster 289: 81 points (0.0%)
  Cluster 290: 22 points (0.0%)
  Cluster 291: 107 points (0.1%)
  Cluster 292: 15 points (0.0%)
  Cluster 293: 33 points (0.0%)
  Cluster 294: 10 points (0.0%)
  Cluster 295: 7 points (0.0%)
  Cluster 296: 13 points (0.0%)
  Cluster 297: 14 points (0.0%)
  Cluster 298: 21 points (0.0%)
  Cluster 299: 12 points (0.0%)
  Cluster 300: 20 points (0.0%)
  Cluster 301: 22 points (0.0%)
  Cluster 302: 57 points (0.0%)
  Cluster 303: 64 points (0.0%)
  Cluster 304: 10 points (0.0%)
  Cluster 305: 44 points (0.0%)
  Cluster 306: 17 points (0.0%)
  Cluster 307: 16 points (0.0%)
  Cluster 308: 22 points (0.0%)
  Cluster 309: 13 points (0.0%)
  Cluster 310: 12 points (0.0%)
  Cluster 311: 18 points (0.0%)
  Cluster 312: 30 points (0.0%)
  Cluster 313: 17 points (0.0%)
  Cluster 314: 18 points (0.0%)
  Cluster 315: 23 points (0.0%)
  Cluster 316: 117 points (0.1%)
  Cluster 317: 717 points (0.4%)
  Cluster 318: 18 points (0.0%)
  Cluster 319: 26 points (0.0%)
  Cluster 320: 82 points (0.0%)
  Cluster 321: 11 points (0.0%)
  Cluster 322: 32 points (0.0%)
  Cluster 323: 36 points (0.0%)
  Cluster 324: 312 points (0.2%)
  Cluster 325: 53 points (0.0%)
  Cluster 326: 11 points (0.0%)
  Cluster 327: 23 points (0.0%)
  Cluster 328: 22 points (0.0%)
  Cluster 329: 73 points (0.0%)
  Cluster 330: 24 points (0.0%)
  Cluster 331: 26 points (0.0%)
  Cluster 332: 6 points (0.0%)
  Cluster 333: 18 points (0.0%)
  Cluster 334: 48 points (0.0%)
  Cluster 335: 53 points (0.0%)
  Cluster 336: 11 points (0.0%)
  Cluster 337: 44 points (0.0%)
  Cluster 338: 15 points (0.0%)
  Cluster 339: 26 points (0.0%)
  Cluster 340: 16 points (0.0%)
  Cluster 341: 10 points (0.0%)
  Cluster 342: 42 points (0.0%)
  Cluster 343: 11 points (0.0%)
  Cluster 344: 74 points (0.0%)
  Cluster 345: 29 points (0.0%)
  Cluster 346: 17 points (0.0%)
  Cluster 347: 20 points (0.0%)
  Cluster 348: 27 points (0.0%)
  Cluster 349: 17 points (0.0%)
  Cluster 350: 13 points (0.0%)
  Cluster 351: 10 points (0.0%)
  Cluster 352: 10 points (0.0%)
  Cluster 353: 19 points (0.0%)
  Cluster 354: 25 points (0.0%)
  Cluster 355: 12 points (0.0%)
  Cluster 356: 10 points (0.0%)
  Cluster 357: 14 points (0.0%)
  Cluster 358: 11 points (0.0%)
  Cluster 359: 13 points (0.0%)
  Cluster 360: 33 points (0.0%)
  Cluster 361: 16 points (0.0%)
  Cluster 362: 18 points (0.0%)
  Cluster 363: 24 points (0.0%)
  Cluster 364: 28 points (0.0%)
  Cluster 365: 16 points (0.0%)
  Cluster 366: 10 points (0.0%)
  Cluster 367: 20 points (0.0%)
  Cluster 368: 11 points (0.0%)
  Cluster 369: 39 points (0.0%)
  Cluster 370: 18 points (0.0%)
  Cluster 371: 11 points (0.0%)
  Cluster 372: 16 points (0.0%)
  Cluster 373: 14 points (0.0%)
  Cluster 374: 11 points (0.0%)
  Cluster 375: 10 points (0.0%)
  Cluster 376: 28 points (0.0%)
  Cluster 377: 11 points (0.0%)
  Cluster 378: 21 points (0.0%)
  Cluster 379: 13 points (0.0%)
  Cluster 380: 10 points (0.0%)
  Cluster 381: 10 points (0.0%)
  Cluster 382: 44 points (0.0%)
  Cluster 383: 202 points (0.1%)
  Cluster 384: 160 points (0.1%)
  Cluster 385: 53 points (0.0%)
  Cluster 386: 13 points (0.0%)
  Cluster 387: 10 points (0.0%)
  Cluster 388: 17 points (0.0%)
  Cluster 389: 19 points (0.0%)
  Cluster 390: 13 points (0.0%)
  Cluster 391: 19 points (0.0%)
  Cluster 392: 20 points (0.0%)
  Cluster 393: 14 points (0.0%)
  Cluster 394: 13 points (0.0%)
  Cluster 395: 15 points (0.0%)
  Cluster 396: 11 points (0.0%)
  Cluster 397: 27 points (0.0%)
  Cluster 398: 16 points (0.0%)
  Cluster 399: 19 points (0.0%)
  Cluster 400: 11 points (0.0%)
  Cluster 401: 13 points (0.0%)
  Cluster 402: 8 points (0.0%)
  Cluster 403: 16 points (0.0%)
  Cluster 404: 10 points (0.0%)
  Cluster 405: 33 points (0.0%)
  Cluster 406: 10 points (0.0%)
  Cluster 407: 22 points (0.0%)
  Cluster 408: 10 points (0.0%)
  Cluster 409: 11 points (0.0%)
  Cluster 410: 37 points (0.0%)
  Cluster 411: 13 points (0.0%)
  Cluster 412: 12 points (0.0%)
  Cluster 413: 10 points (0.0%)
  Cluster 414: 20 points (0.0%)
  Cluster 415: 13 points (0.0%)
  Cluster 416: 13 points (0.0%)
  Cluster 417: 12 points (0.0%)
  Cluster 418: 24 points (0.0%)
  Cluster 419: 13 points (0.0%)
  Cluster 420: 16 points (0.0%)
  Cluster 421: 9 points (0.0%)
  Cluster 422: 11 points (0.0%)
  Cluster 423: 11 points (0.0%)
  Cluster 424: 10 points (0.0%)
  Cluster 425: 12 points (0.0%)
  Cluster 426: 12 points (0.0%)
  Cluster 427: 10 points (0.0%)
  Cluster 428: 10 points (0.0%)
  Cluster 429: 11 points (0.0%)
  Cluster 430: 26 points (0.0%)
  Cluster 431: 13 points (0.0%)
  Cluster 432: 18 points (0.0%)
  Cluster 433: 12 points (0.0%)
  Cluster 434: 33 points (0.0%)
  Cluster 435: 15 points (0.0%)
  Cluster 436: 16 points (0.0%)
  Cluster 437: 12 points (0.0%)
  Cluster 438: 10 points (0.0%)
  Cluster 439: 27 points (0.0%)
  Cluster 440: 12 points (0.0%)
  Cluster 441: 7 points (0.0%)
  Cluster 442: 16 points (0.0%)
  Cluster 443: 13 points (0.0%)
  Cluster 444: 11 points (0.0%)
  Cluster 445: 19 points (0.0%)
  Cluster 446: 18 points (0.0%)
  Cluster 447: 18 points (0.0%)
  Cluster 448: 11 points (0.0%)
  Cluster 449: 17 points (0.0%)
  Cluster 450: 12 points (0.0%)
  Cluster 451: 10 points (0.0%)

Generating visualization...

================================================================================

================================================================================
STEP 4: SUPERVISED LEARNING (XGBOOST)
================================================================================
================================================================================
FEATURE PREPARATION FOR MACHINE LEARNING
================================================================================

1. Removing 8 non-feature columns:
   - Flow ID
   - Source IP
   - Destination IP
   - Timestamp
   - Label
   - Binary_Label
   - Source Port
   - Destination Port

2. Ensuring all features are numeric...
   ✓ All features are numeric

3. Handling infinite values...
   Found 216 infinite values
   ✓ Replaced with NaN

4. Handling missing values...
   Found 244 missing values
   Imputing with column medians...
   ✓ Missing values imputed

================================================================================
FINAL DATASET FOR MODELING
================================================================================

Samples: 191,033
Features: 78

Class distribution:
  Class 0 (BENIGN): 189,067 (98.97%)
  Class 1 (ATTACK): 1,966 (1.03%)

Feature types:
  Port features: 0
  Other features: 78

Sample feature names:
  1. Protocol
  2. Flow Duration
  3. Total Fwd Packets
  4. Total Backward Packets
  5. Total Length of Fwd Packets
  6. Total Length of Bwd Packets
  7. Fwd Packet Length Max
  8. Fwd Packet Length Min
  9. Fwd Packet Length Mean
  10. Fwd Packet Length Std
  ... and 68 more

================================================================================
================================================================================
MODEL 3: XGBOOST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Calculated scale_pos_weight: 96.17
  (Ratio of negative to positive samples)

Creating XGBoost model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 4.32 seconds

================================================================================
XGBOOST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  0.9994 (+/- 0.0001)
  Train: 0.9996 (+/- 0.0001)

PRECISION:
  Test:  0.9524 (+/- 0.0114)
  Train: 0.9606 (+/- 0.0058)

RECALL:
  Test:  0.9944 (+/- 0.0036)
  Train: 1.0000 (+/- 0.0000)

F1:
  Test:  0.9729 (+/- 0.0056)
  Train: 0.9799 (+/- 0.0030)

ROC_AUC:
  Test:  0.9999 (+/- 0.0001)
  Train: 1.0000 (+/- 0.0000)

================================================================================

================================================================================
✓ FILE 2/7 COMPLETED SUCCESSFULLY
================================================================================

✓ [2/7] Friday-WorkingHours-Morning.pcap_ISCX.csv - SUCCESS

################################################################################
# PROCESSING FILE 3/7: Monday-WorkingHours.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================
✓ Data loaded successfully!
  Rows: 529,918
  Columns: 85
✓ Column names cleaned (whitespace stripped)

================================================================================
STEP 2: CREATING BINARY LABELS
================================================================================
================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  BENIGN: 529,918 (100.00%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0

✓ Binary labels created in 'Binary_Label' column
================================================================================

================================================================================
STEP 3: UNSUPERVISED LEARNING (DBSCAN)
================================================================================
================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 529,918

2. Handling infinite values...
   Replaced 810 infinite values with NaN

3. Handling missing values...
   Imputed 874 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 22 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Fwd IAT Total', 'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Min', 'Fwd Packets/s', 'Packet Length Std', 'SYN Flag Count']

5. Features after correlation removal: 58

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 529,918 samples × 58 features
================================================================================
================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 58

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 91.30%

✓ PCA complete!
================================================================================
================================================================================
APPLYING DBSCAN CLUSTERING
================================================================================

Parameters:
  eps = 0.5
  min_samples = 10

Clustering 529,918 samples with 20 features...

✓ Clustering complete in 162.44 seconds

Results:
  Clusters found: 679
  Noise points: 42,563 (8.0%)
  Clustered points: 487,355 (92.0%)

Cluster size distribution:
  Cluster 0: 14 points (0.0%)
  Cluster 1: 33 points (0.0%)
  Cluster 2: 536 points (0.1%)
  Cluster 3: 12 points (0.0%)
  Cluster 4: 10 points (0.0%)
  Cluster 5: 107 points (0.0%)
  Cluster 6: 45,371 points (8.6%)
  Cluster 7: 1,944 points (0.4%)
  Cluster 8: 8,987 points (1.7%)
  Cluster 9: 122 points (0.0%)
  Cluster 10: 29 points (0.0%)
  Cluster 11: 181 points (0.0%)
  Cluster 12: 57 points (0.0%)
  Cluster 13: 25,467 points (4.8%)
  Cluster 14: 17 points (0.0%)
  Cluster 15: 18,700 points (3.5%)
  Cluster 16: 1,479 points (0.3%)
  Cluster 17: 1,013 points (0.2%)
  Cluster 18: 11 points (0.0%)
  Cluster 19: 169 points (0.0%)
  Cluster 20: 3,708 points (0.7%)
  Cluster 21: 10 points (0.0%)
  Cluster 22: 11 points (0.0%)
  Cluster 23: 10 points (0.0%)
  Cluster 24: 1,447 points (0.3%)
  Cluster 25: 28 points (0.0%)
  Cluster 26: 24 points (0.0%)
  Cluster 27: 111 points (0.0%)
  Cluster 28: 24,186 points (4.6%)
  Cluster 29: 401 points (0.1%)
  Cluster 30: 90 points (0.0%)
  Cluster 31: 4,784 points (0.9%)
  Cluster 32: 665 points (0.1%)
  Cluster 33: 5,441 points (1.0%)
  Cluster 34: 483 points (0.1%)
  Cluster 35: 101 points (0.0%)
  Cluster 36: 502 points (0.1%)
  Cluster 37: 205,790 points (38.8%)
  Cluster 38: 16 points (0.0%)
  Cluster 39: 24 points (0.0%)
  Cluster 40: 52 points (0.0%)
  Cluster 41: 47 points (0.0%)
  Cluster 42: 316 points (0.1%)
  Cluster 43: 4,588 points (0.9%)
  Cluster 44: 7,359 points (1.4%)
  Cluster 45: 10,407 points (2.0%)
  Cluster 46: 53 points (0.0%)
  Cluster 47: 30 points (0.0%)
  Cluster 48: 131 points (0.0%)
  Cluster 49: 150 points (0.0%)
  Cluster 50: 70 points (0.0%)
  Cluster 51: 2,971 points (0.6%)
  Cluster 52: 48 points (0.0%)
  Cluster 53: 18 points (0.0%)
  Cluster 54: 95 points (0.0%)
  Cluster 55: 13 points (0.0%)
  Cluster 56: 8,920 points (1.7%)
  Cluster 57: 334 points (0.1%)
  Cluster 58: 35 points (0.0%)
  Cluster 59: 12 points (0.0%)
  Cluster 60: 590 points (0.1%)
  Cluster 61: 12 points (0.0%)
  Cluster 62: 391 points (0.1%)
  Cluster 63: 25 points (0.0%)
  Cluster 64: 33 points (0.0%)
  Cluster 65: 455 points (0.1%)
  Cluster 66: 50 points (0.0%)
  Cluster 67: 62 points (0.0%)
  Cluster 68: 270 points (0.1%)
  Cluster 69: 186 points (0.0%)
  Cluster 70: 193 points (0.0%)
  Cluster 71: 477 points (0.1%)
  Cluster 72: 120 points (0.0%)
  Cluster 73: 68 points (0.0%)
  Cluster 74: 62 points (0.0%)
  Cluster 75: 39 points (0.0%)
  Cluster 76: 2,523 points (0.5%)
  Cluster 77: 733 points (0.1%)
  Cluster 78: 11 points (0.0%)
  Cluster 79: 130 points (0.0%)
  Cluster 80: 40 points (0.0%)
  Cluster 81: 265 points (0.1%)
  Cluster 82: 49 points (0.0%)
  Cluster 83: 388 points (0.1%)
  Cluster 84: 70 points (0.0%)
  Cluster 85: 19 points (0.0%)
  Cluster 86: 47 points (0.0%)
  Cluster 87: 1,670 points (0.3%)
  Cluster 88: 339 points (0.1%)
  Cluster 89: 32 points (0.0%)
  Cluster 90: 317 points (0.1%)
  Cluster 91: 292 points (0.1%)
  Cluster 92: 722 points (0.1%)
  Cluster 93: 13 points (0.0%)
  Cluster 94: 65 points (0.0%)
  Cluster 95: 9,181 points (1.7%)
  Cluster 96: 62 points (0.0%)
  Cluster 97: 450 points (0.1%)
  Cluster 98: 45 points (0.0%)
  Cluster 99: 192 points (0.0%)
  Cluster 100: 23 points (0.0%)
  Cluster 101: 1,310 points (0.2%)
  Cluster 102: 3,339 points (0.6%)
  Cluster 103: 30 points (0.0%)
  Cluster 104: 355 points (0.1%)
  Cluster 105: 21 points (0.0%)
  Cluster 106: 68 points (0.0%)
  Cluster 107: 11 points (0.0%)
  Cluster 108: 413 points (0.1%)
  Cluster 109: 1,584 points (0.3%)
  Cluster 110: 221 points (0.0%)
  Cluster 111: 16 points (0.0%)
  Cluster 112: 315 points (0.1%)
  Cluster 113: 163 points (0.0%)
  Cluster 114: 10 points (0.0%)
  Cluster 115: 736 points (0.1%)
  Cluster 116: 291 points (0.1%)
  Cluster 117: 209 points (0.0%)
  Cluster 118: 645 points (0.1%)
  Cluster 119: 255 points (0.0%)
  Cluster 120: 40 points (0.0%)
  Cluster 121: 25 points (0.0%)
  Cluster 122: 322 points (0.1%)
  Cluster 123: 53 points (0.0%)
  Cluster 124: 172 points (0.0%)
  Cluster 125: 10 points (0.0%)
  Cluster 126: 243 points (0.0%)
  Cluster 127: 347 points (0.1%)
  Cluster 128: 30 points (0.0%)
  Cluster 129: 15 points (0.0%)
  Cluster 130: 189 points (0.0%)
  Cluster 131: 29 points (0.0%)
  Cluster 132: 554 points (0.1%)
  Cluster 133: 27 points (0.0%)
  Cluster 134: 23 points (0.0%)
  Cluster 135: 70 points (0.0%)
  Cluster 136: 11 points (0.0%)
  Cluster 137: 17 points (0.0%)
  Cluster 138: 57 points (0.0%)
  Cluster 139: 136 points (0.0%)
  Cluster 140: 15 points (0.0%)
  Cluster 141: 14 points (0.0%)
  Cluster 142: 90 points (0.0%)
  Cluster 143: 23 points (0.0%)
  Cluster 144: 16 points (0.0%)
  Cluster 145: 266 points (0.1%)
  Cluster 146: 414 points (0.1%)
  Cluster 147: 6,700 points (1.3%)
  Cluster 148: 4,494 points (0.8%)
  Cluster 149: 10,770 points (2.0%)
  Cluster 150: 135 points (0.0%)
  Cluster 151: 44 points (0.0%)
  Cluster 152: 21 points (0.0%)
  Cluster 153: 12 points (0.0%)
  Cluster 154: 442 points (0.1%)
  Cluster 155: 18 points (0.0%)
  Cluster 156: 113 points (0.0%)
  Cluster 157: 57 points (0.0%)
  Cluster 158: 44 points (0.0%)
  Cluster 159: 27 points (0.0%)
  Cluster 160: 43 points (0.0%)
  Cluster 161: 93 points (0.0%)
  Cluster 162: 42 points (0.0%)
  Cluster 163: 42 points (0.0%)
  Cluster 164: 25 points (0.0%)
  Cluster 165: 18 points (0.0%)
  Cluster 166: 674 points (0.1%)
  Cluster 167: 235 points (0.0%)
  Cluster 168: 35 points (0.0%)
  Cluster 169: 36 points (0.0%)
  Cluster 170: 18 points (0.0%)
  Cluster 171: 7 points (0.0%)
  Cluster 172: 1,995 points (0.4%)
  Cluster 173: 196 points (0.0%)
  Cluster 174: 19 points (0.0%)
  Cluster 175: 61 points (0.0%)
  Cluster 176: 70 points (0.0%)
  Cluster 177: 11 points (0.0%)
  Cluster 178: 73 points (0.0%)
  Cluster 179: 21 points (0.0%)
  Cluster 180: 167 points (0.0%)
  Cluster 181: 14 points (0.0%)
  Cluster 182: 18 points (0.0%)
  Cluster 183: 211 points (0.0%)
  Cluster 184: 120 points (0.0%)
  Cluster 185: 724 points (0.1%)
  Cluster 186: 12 points (0.0%)
  Cluster 187: 70 points (0.0%)
  Cluster 188: 14 points (0.0%)
  Cluster 189: 11 points (0.0%)
  Cluster 190: 47 points (0.0%)
  Cluster 191: 11 points (0.0%)
  Cluster 192: 27 points (0.0%)
  Cluster 193: 27 points (0.0%)
  Cluster 194: 17 points (0.0%)
  Cluster 195: 11 points (0.0%)
  Cluster 196: 12 points (0.0%)
  Cluster 197: 10 points (0.0%)
  Cluster 198: 17 points (0.0%)
  Cluster 199: 26 points (0.0%)
  Cluster 200: 20 points (0.0%)
  Cluster 201: 30 points (0.0%)
  Cluster 202: 30 points (0.0%)
  Cluster 203: 22 points (0.0%)
  Cluster 204: 25 points (0.0%)
  Cluster 205: 11 points (0.0%)
  Cluster 206: 135 points (0.0%)
  Cluster 207: 52 points (0.0%)
  Cluster 208: 22 points (0.0%)
  Cluster 209: 134 points (0.0%)
  Cluster 210: 1,993 points (0.4%)
  Cluster 211: 18 points (0.0%)
  Cluster 212: 54 points (0.0%)
  Cluster 213: 112 points (0.0%)
  Cluster 214: 18 points (0.0%)
  Cluster 215: 34 points (0.0%)
  Cluster 216: 57 points (0.0%)
  Cluster 217: 10 points (0.0%)
  Cluster 218: 11 points (0.0%)
  Cluster 219: 39 points (0.0%)
  Cluster 220: 33 points (0.0%)
  Cluster 221: 52 points (0.0%)
  Cluster 222: 47 points (0.0%)
  Cluster 223: 34 points (0.0%)
  Cluster 224: 22 points (0.0%)
  Cluster 225: 281 points (0.1%)
  Cluster 226: 28 points (0.0%)
  Cluster 227: 14 points (0.0%)
  Cluster 228: 58 points (0.0%)
  Cluster 229: 63 points (0.0%)
  Cluster 230: 57 points (0.0%)
  Cluster 231: 83 points (0.0%)
  Cluster 232: 12 points (0.0%)
  Cluster 233: 28 points (0.0%)
  Cluster 234: 25 points (0.0%)
  Cluster 235: 513 points (0.1%)
  Cluster 236: 32 points (0.0%)
  Cluster 237: 78 points (0.0%)
  Cluster 238: 43 points (0.0%)
  Cluster 239: 53 points (0.0%)
  Cluster 240: 46 points (0.0%)
  Cluster 241: 24 points (0.0%)
  Cluster 242: 17 points (0.0%)
  Cluster 243: 76 points (0.0%)
  Cluster 244: 88 points (0.0%)
  Cluster 245: 42 points (0.0%)
  Cluster 246: 12 points (0.0%)
  Cluster 247: 11 points (0.0%)
  Cluster 248: 16 points (0.0%)
  Cluster 249: 45 points (0.0%)
  Cluster 250: 29 points (0.0%)
  Cluster 251: 42 points (0.0%)
  Cluster 252: 66 points (0.0%)
  Cluster 253: 24 points (0.0%)
  Cluster 254: 22 points (0.0%)
  Cluster 255: 33 points (0.0%)
  Cluster 256: 1,747 points (0.3%)
  Cluster 257: 30 points (0.0%)
  Cluster 258: 9,689 points (1.8%)
  Cluster 259: 909 points (0.2%)
  Cluster 260: 83 points (0.0%)
  Cluster 261: 12 points (0.0%)
  Cluster 262: 25 points (0.0%)
  Cluster 263: 646 points (0.1%)
  Cluster 264: 19 points (0.0%)
  Cluster 265: 13 points (0.0%)
  Cluster 266: 19 points (0.0%)
  Cluster 267: 27 points (0.0%)
  Cluster 268: 44 points (0.0%)
  Cluster 269: 52 points (0.0%)
  Cluster 270: 15 points (0.0%)
  Cluster 271: 47 points (0.0%)
  Cluster 272: 11 points (0.0%)
  Cluster 273: 51 points (0.0%)
  Cluster 274: 36 points (0.0%)
  Cluster 275: 34 points (0.0%)
  Cluster 276: 28 points (0.0%)
  Cluster 277: 46 points (0.0%)
  Cluster 278: 43 points (0.0%)
  Cluster 279: 10 points (0.0%)
  Cluster 280: 13 points (0.0%)
  Cluster 281: 12 points (0.0%)
  Cluster 282: 73 points (0.0%)
  Cluster 283: 38 points (0.0%)
  Cluster 284: 36 points (0.0%)
  Cluster 285: 10 points (0.0%)
  Cluster 286: 205 points (0.0%)
  Cluster 287: 15 points (0.0%)
  Cluster 288: 24 points (0.0%)
  Cluster 289: 18 points (0.0%)
  Cluster 290: 15 points (0.0%)
  Cluster 291: 11 points (0.0%)
  Cluster 292: 50 points (0.0%)
  Cluster 293: 21 points (0.0%)
  Cluster 294: 13 points (0.0%)
  Cluster 295: 15 points (0.0%)
  Cluster 296: 25 points (0.0%)
  Cluster 297: 17 points (0.0%)
  Cluster 298: 11 points (0.0%)
  Cluster 299: 48 points (0.0%)
  Cluster 300: 30 points (0.0%)
  Cluster 301: 20 points (0.0%)
  Cluster 302: 17 points (0.0%)
  Cluster 303: 15 points (0.0%)
  Cluster 304: 23 points (0.0%)
  Cluster 305: 40 points (0.0%)
  Cluster 306: 18 points (0.0%)
  Cluster 307: 23 points (0.0%)
  Cluster 308: 37 points (0.0%)
  Cluster 309: 31 points (0.0%)
  Cluster 310: 21 points (0.0%)
  Cluster 311: 14 points (0.0%)
  Cluster 312: 17 points (0.0%)
  Cluster 313: 17 points (0.0%)
  Cluster 314: 81 points (0.0%)
  Cluster 315: 13 points (0.0%)
  Cluster 316: 17 points (0.0%)
  Cluster 317: 28 points (0.0%)
  Cluster 318: 43 points (0.0%)
  Cluster 319: 31 points (0.0%)
  Cluster 320: 14 points (0.0%)
  Cluster 321: 9 points (0.0%)
  Cluster 322: 20 points (0.0%)
  Cluster 323: 7 points (0.0%)
  Cluster 324: 93 points (0.0%)
  Cluster 325: 14 points (0.0%)
  Cluster 326: 11 points (0.0%)
  Cluster 327: 14 points (0.0%)
  Cluster 328: 32 points (0.0%)
  Cluster 329: 19 points (0.0%)
  Cluster 330: 26 points (0.0%)
  Cluster 331: 43 points (0.0%)
  Cluster 332: 66 points (0.0%)
  Cluster 333: 29 points (0.0%)
  Cluster 334: 10 points (0.0%)
  Cluster 335: 11 points (0.0%)
  Cluster 336: 12 points (0.0%)
  Cluster 337: 18 points (0.0%)
  Cluster 338: 18 points (0.0%)
  Cluster 339: 15 points (0.0%)
  Cluster 340: 14 points (0.0%)
  Cluster 341: 60 points (0.0%)
  Cluster 342: 32 points (0.0%)
  Cluster 343: 13 points (0.0%)
  Cluster 344: 19 points (0.0%)
  Cluster 345: 26 points (0.0%)
  Cluster 346: 17 points (0.0%)
  Cluster 347: 11 points (0.0%)
  Cluster 348: 10 points (0.0%)
  Cluster 349: 93 points (0.0%)
  Cluster 350: 88 points (0.0%)
  Cluster 351: 109 points (0.0%)
  Cluster 352: 19 points (0.0%)
  Cluster 353: 52 points (0.0%)
  Cluster 354: 397 points (0.1%)
  Cluster 355: 10 points (0.0%)
  Cluster 356: 10 points (0.0%)
  Cluster 357: 40 points (0.0%)
  Cluster 358: 13 points (0.0%)
  Cluster 359: 13 points (0.0%)
  Cluster 360: 80 points (0.0%)
  Cluster 361: 15 points (0.0%)
  Cluster 362: 22 points (0.0%)
  Cluster 363: 206 points (0.0%)
  Cluster 364: 241 points (0.0%)
  Cluster 365: 945 points (0.2%)
  Cluster 366: 15 points (0.0%)
  Cluster 367: 14 points (0.0%)
  Cluster 368: 81 points (0.0%)
  Cluster 369: 68 points (0.0%)
  Cluster 370: 15 points (0.0%)
  Cluster 371: 68 points (0.0%)
  Cluster 372: 24 points (0.0%)
  Cluster 373: 14 points (0.0%)
  Cluster 374: 69 points (0.0%)
  Cluster 375: 22 points (0.0%)
  Cluster 376: 14 points (0.0%)
  Cluster 377: 329 points (0.1%)
  Cluster 378: 14 points (0.0%)
  Cluster 379: 13 points (0.0%)
  Cluster 380: 328 points (0.1%)
  Cluster 381: 16 points (0.0%)
  Cluster 382: 14 points (0.0%)
  Cluster 383: 30 points (0.0%)
  Cluster 384: 204 points (0.0%)
  Cluster 385: 26 points (0.0%)
  Cluster 386: 11 points (0.0%)
  Cluster 387: 15 points (0.0%)
  Cluster 388: 11 points (0.0%)
  Cluster 389: 20 points (0.0%)
  Cluster 390: 10 points (0.0%)
  Cluster 391: 82 points (0.0%)
  Cluster 392: 22 points (0.0%)
  Cluster 393: 15 points (0.0%)
  Cluster 394: 22 points (0.0%)
  Cluster 395: 21 points (0.0%)
  Cluster 396: 35 points (0.0%)
  Cluster 397: 27 points (0.0%)
  Cluster 398: 18 points (0.0%)
  Cluster 399: 11 points (0.0%)
  Cluster 400: 21 points (0.0%)
  Cluster 401: 15 points (0.0%)
  Cluster 402: 16 points (0.0%)
  Cluster 403: 19 points (0.0%)
  Cluster 404: 10 points (0.0%)
  Cluster 405: 11 points (0.0%)
  Cluster 406: 24 points (0.0%)
  Cluster 407: 72 points (0.0%)
  Cluster 408: 17 points (0.0%)
  Cluster 409: 35 points (0.0%)
  Cluster 410: 10 points (0.0%)
  Cluster 411: 11 points (0.0%)
  Cluster 412: 16 points (0.0%)
  Cluster 413: 19 points (0.0%)
  Cluster 414: 22 points (0.0%)
  Cluster 415: 16 points (0.0%)
  Cluster 416: 16 points (0.0%)
  Cluster 417: 16 points (0.0%)
  Cluster 418: 17 points (0.0%)
  Cluster 419: 21 points (0.0%)
  Cluster 420: 10 points (0.0%)
  Cluster 421: 11 points (0.0%)
  Cluster 422: 11 points (0.0%)
  Cluster 423: 23 points (0.0%)
  Cluster 424: 84 points (0.0%)
  Cluster 425: 11 points (0.0%)
  Cluster 426: 10 points (0.0%)
  Cluster 427: 22 points (0.0%)
  Cluster 428: 14 points (0.0%)
  Cluster 429: 12 points (0.0%)
  Cluster 430: 7 points (0.0%)
  Cluster 431: 34 points (0.0%)
  Cluster 432: 43 points (0.0%)
  Cluster 433: 19 points (0.0%)
  Cluster 434: 78 points (0.0%)
  Cluster 435: 17 points (0.0%)
  Cluster 436: 23 points (0.0%)
  Cluster 437: 15 points (0.0%)
  Cluster 438: 16 points (0.0%)
  Cluster 439: 18 points (0.0%)
  Cluster 440: 12 points (0.0%)
  Cluster 441: 41 points (0.0%)
  Cluster 442: 72 points (0.0%)
  Cluster 443: 13 points (0.0%)
  Cluster 444: 381 points (0.1%)
  Cluster 445: 17 points (0.0%)
  Cluster 446: 148 points (0.0%)
  Cluster 447: 119 points (0.0%)
  Cluster 448: 326 points (0.1%)
  Cluster 449: 19 points (0.0%)
  Cluster 450: 966 points (0.2%)
  Cluster 451: 15 points (0.0%)
  Cluster 452: 52 points (0.0%)
  Cluster 453: 296 points (0.1%)
  Cluster 454: 25 points (0.0%)
  Cluster 455: 11 points (0.0%)
  Cluster 456: 10 points (0.0%)
  Cluster 457: 12 points (0.0%)
  Cluster 458: 21 points (0.0%)
  Cluster 459: 17 points (0.0%)
  Cluster 460: 15 points (0.0%)
  Cluster 461: 14 points (0.0%)
  Cluster 462: 17 points (0.0%)
  Cluster 463: 15 points (0.0%)
  Cluster 464: 13 points (0.0%)
  Cluster 465: 21 points (0.0%)
  Cluster 466: 21 points (0.0%)
  Cluster 467: 23 points (0.0%)
  Cluster 468: 11 points (0.0%)
  Cluster 469: 232 points (0.0%)
  Cluster 470: 11 points (0.0%)
  Cluster 471: 54 points (0.0%)
  Cluster 472: 15 points (0.0%)
  Cluster 473: 64 points (0.0%)
  Cluster 474: 20 points (0.0%)
  Cluster 475: 15 points (0.0%)
  Cluster 476: 10 points (0.0%)
  Cluster 477: 25 points (0.0%)
  Cluster 478: 12 points (0.0%)
  Cluster 479: 17 points (0.0%)
  Cluster 480: 46 points (0.0%)
  Cluster 481: 14 points (0.0%)
  Cluster 482: 10 points (0.0%)
  Cluster 483: 11 points (0.0%)
  Cluster 484: 41 points (0.0%)
  Cluster 485: 15 points (0.0%)
  Cluster 486: 15 points (0.0%)
  Cluster 487: 10 points (0.0%)
  Cluster 488: 1,167 points (0.2%)
  Cluster 489: 10 points (0.0%)
  Cluster 490: 10 points (0.0%)
  Cluster 491: 11 points (0.0%)
  Cluster 492: 11 points (0.0%)
  Cluster 493: 16 points (0.0%)
  Cluster 494: 17 points (0.0%)
  Cluster 495: 20 points (0.0%)
  Cluster 496: 11 points (0.0%)
  Cluster 497: 83 points (0.0%)
  Cluster 498: 32 points (0.0%)
  Cluster 499: 22 points (0.0%)
  Cluster 500: 12 points (0.0%)
  Cluster 501: 52 points (0.0%)
  Cluster 502: 16 points (0.0%)
  Cluster 503: 82 points (0.0%)
  Cluster 504: 10 points (0.0%)
  Cluster 505: 14 points (0.0%)
  Cluster 506: 10 points (0.0%)
  Cluster 507: 27 points (0.0%)
  Cluster 508: 17 points (0.0%)
  Cluster 509: 26 points (0.0%)
  Cluster 510: 91 points (0.0%)
  Cluster 511: 27 points (0.0%)
  Cluster 512: 12 points (0.0%)
  Cluster 513: 69 points (0.0%)
  Cluster 514: 17 points (0.0%)
  Cluster 515: 16 points (0.0%)
  Cluster 516: 9 points (0.0%)
  Cluster 517: 7 points (0.0%)
  Cluster 518: 28 points (0.0%)
  Cluster 519: 14 points (0.0%)
  Cluster 520: 15 points (0.0%)
  Cluster 521: 17 points (0.0%)
  Cluster 522: 24 points (0.0%)
  Cluster 523: 10 points (0.0%)
  Cluster 524: 22 points (0.0%)
  Cluster 525: 10 points (0.0%)
  Cluster 526: 12 points (0.0%)
  Cluster 527: 10 points (0.0%)
  Cluster 528: 54 points (0.0%)
  Cluster 529: 10 points (0.0%)
  Cluster 530: 16 points (0.0%)
  Cluster 531: 20 points (0.0%)
  Cluster 532: 13 points (0.0%)
  Cluster 533: 13 points (0.0%)
  Cluster 534: 14 points (0.0%)
  Cluster 535: 6 points (0.0%)
  Cluster 536: 20 points (0.0%)
  Cluster 537: 10 points (0.0%)
  Cluster 538: 32 points (0.0%)
  Cluster 539: 12 points (0.0%)
  Cluster 540: 14 points (0.0%)
  Cluster 541: 11 points (0.0%)
  Cluster 542: 10 points (0.0%)
  Cluster 543: 12 points (0.0%)
  Cluster 544: 11 points (0.0%)
  Cluster 545: 61 points (0.0%)
  Cluster 546: 9 points (0.0%)
  Cluster 547: 58 points (0.0%)
  Cluster 548: 10 points (0.0%)
  Cluster 549: 11 points (0.0%)
  Cluster 550: 10 points (0.0%)
  Cluster 551: 10 points (0.0%)
  Cluster 552: 74 points (0.0%)
  Cluster 553: 10 points (0.0%)
  Cluster 554: 12 points (0.0%)
  Cluster 555: 10 points (0.0%)
  Cluster 556: 9 points (0.0%)
  Cluster 557: 10 points (0.0%)
  Cluster 558: 16 points (0.0%)
  Cluster 559: 16 points (0.0%)
  Cluster 560: 6 points (0.0%)
  Cluster 561: 21 points (0.0%)
  Cluster 562: 11 points (0.0%)
  Cluster 563: 10 points (0.0%)
  Cluster 564: 12 points (0.0%)
  Cluster 565: 9 points (0.0%)
  Cluster 566: 30 points (0.0%)
  Cluster 567: 25 points (0.0%)
  Cluster 568: 14 points (0.0%)
  Cluster 569: 11 points (0.0%)
  Cluster 570: 20 points (0.0%)
  Cluster 571: 380 points (0.1%)
  Cluster 572: 10 points (0.0%)
  Cluster 573: 18 points (0.0%)
  Cluster 574: 21 points (0.0%)
  Cluster 575: 12 points (0.0%)
  Cluster 576: 26 points (0.0%)
  Cluster 577: 10 points (0.0%)
  Cluster 578: 13 points (0.0%)
  Cluster 579: 4 points (0.0%)
  Cluster 580: 12 points (0.0%)
  Cluster 581: 4 points (0.0%)
  Cluster 582: 12 points (0.0%)
  Cluster 583: 10 points (0.0%)
  Cluster 584: 10 points (0.0%)
  Cluster 585: 59 points (0.0%)
  Cluster 586: 11 points (0.0%)
  Cluster 587: 11 points (0.0%)
  Cluster 588: 13 points (0.0%)
  Cluster 589: 14 points (0.0%)
  Cluster 590: 10 points (0.0%)
  Cluster 591: 10 points (0.0%)
  Cluster 592: 10 points (0.0%)
  Cluster 593: 10 points (0.0%)
  Cluster 594: 10 points (0.0%)
  Cluster 595: 734 points (0.1%)
  Cluster 596: 210 points (0.0%)
  Cluster 597: 279 points (0.1%)
  Cluster 598: 156 points (0.0%)
  Cluster 599: 165 points (0.0%)
  Cluster 600: 31 points (0.0%)
  Cluster 601: 11 points (0.0%)
  Cluster 602: 10 points (0.0%)
  Cluster 603: 119 points (0.0%)
  Cluster 604: 12 points (0.0%)
  Cluster 605: 28 points (0.0%)
  Cluster 606: 163 points (0.0%)
  Cluster 607: 34 points (0.0%)
  Cluster 608: 19 points (0.0%)
  Cluster 609: 219 points (0.0%)
  Cluster 610: 19 points (0.0%)
  Cluster 611: 28 points (0.0%)
  Cluster 612: 205 points (0.0%)
  Cluster 613: 72 points (0.0%)
  Cluster 614: 156 points (0.0%)
  Cluster 615: 111 points (0.0%)
  Cluster 616: 126 points (0.0%)
  Cluster 617: 99 points (0.0%)
  Cluster 618: 229 points (0.0%)
  Cluster 619: 16 points (0.0%)
  Cluster 620: 19 points (0.0%)
  Cluster 621: 331 points (0.1%)
  Cluster 622: 62 points (0.0%)
  Cluster 623: 30 points (0.0%)
  Cluster 624: 125 points (0.0%)
  Cluster 625: 38 points (0.0%)
  Cluster 626: 38 points (0.0%)
  Cluster 627: 48 points (0.0%)
  Cluster 628: 37 points (0.0%)
  Cluster 629: 45 points (0.0%)
  Cluster 630: 12 points (0.0%)
  Cluster 631: 71 points (0.0%)
  Cluster 632: 42 points (0.0%)
  Cluster 633: 111 points (0.0%)
  Cluster 634: 21 points (0.0%)
  Cluster 635: 119 points (0.0%)
  Cluster 636: 29 points (0.0%)
  Cluster 637: 11 points (0.0%)
  Cluster 638: 12 points (0.0%)
  Cluster 639: 12 points (0.0%)
  Cluster 640: 32 points (0.0%)
  Cluster 641: 76 points (0.0%)
  Cluster 642: 21 points (0.0%)
  Cluster 643: 10 points (0.0%)
  Cluster 644: 39 points (0.0%)
  Cluster 645: 28 points (0.0%)
  Cluster 646: 22 points (0.0%)
  Cluster 647: 11 points (0.0%)
  Cluster 648: 22 points (0.0%)
  Cluster 649: 22 points (0.0%)
  Cluster 650: 27 points (0.0%)
  Cluster 651: 26 points (0.0%)
  Cluster 652: 34 points (0.0%)
  Cluster 653: 14 points (0.0%)
  Cluster 654: 12 points (0.0%)
  Cluster 655: 12 points (0.0%)
  Cluster 656: 22 points (0.0%)
  Cluster 657: 20 points (0.0%)
  Cluster 658: 17 points (0.0%)
  Cluster 659: 18 points (0.0%)
  Cluster 660: 17 points (0.0%)
  Cluster 661: 14 points (0.0%)
  Cluster 662: 10 points (0.0%)
  Cluster 663: 11 points (0.0%)
  Cluster 664: 10 points (0.0%)
  Cluster 665: 10 points (0.0%)
  Cluster 666: 11 points (0.0%)
  Cluster 667: 22 points (0.0%)
  Cluster 668: 14 points (0.0%)
  Cluster 669: 13 points (0.0%)
  Cluster 670: 10 points (0.0%)
  Cluster 671: 12 points (0.0%)
  Cluster 672: 11 points (0.0%)
  Cluster 673: 12 points (0.0%)
  Cluster 674: 16 points (0.0%)
  Cluster 675: 18 points (0.0%)
  Cluster 676: 10 points (0.0%)
  Cluster 677: 10 points (0.0%)
  Cluster 678: 14 points (0.0%)

Generating visualization...

================================================================================

================================================================================
STEP 4: SUPERVISED LEARNING (XGBOOST)
================================================================================
================================================================================
FEATURE PREPARATION FOR MACHINE LEARNING
================================================================================

1. Removing 8 non-feature columns:
   - Flow ID
   - Source IP
   - Destination IP
   - Timestamp
   - Label
   - Binary_Label
   - Source Port
   - Destination Port

2. Ensuring all features are numeric...
   ✓ All features are numeric

3. Handling infinite values...
   Found 810 infinite values
   ✓ Replaced with NaN

4. Handling missing values...
   Found 874 missing values
   Imputing with column medians...
   ✓ Missing values imputed

================================================================================
FINAL DATASET FOR MODELING
================================================================================

Samples: 529,918
Features: 78

Class distribution:
  Class 0 (BENIGN): 529,918 (100.00%)
  Class 1 (ATTACK): 0 (0.00%)

Feature types:
  Port features: 0
  Other features: 78

Sample feature names:
  1. Protocol
  2. Flow Duration
  3. Total Fwd Packets
  4. Total Backward Packets
  5. Total Length of Fwd Packets
  6. Total Length of Bwd Packets
  7. Fwd Packet Length Max
  8. Fwd Packet Length Min
  9. Fwd Packet Length Mean
  10. Fwd Packet Length Std
  ... and 68 more

================================================================================
================================================================================
MODEL 3: XGBOOST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Calculated scale_pos_weight: inf
  (Ratio of negative to positive samples)

Creating XGBoost model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 6.05 seconds

================================================================================
XGBOOST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

PRECISION:
  Test:  0.0000 (+/- 0.0000)
  Train: 0.0000 (+/- 0.0000)

RECALL:
  Test:  0.0000 (+/- 0.0000)
  Train: 0.0000 (+/- 0.0000)

F1:
  Test:  0.0000 (+/- 0.0000)
  Train: 0.0000 (+/- 0.0000)

ROC_AUC:
  Test:  nan (+/- nan)
  Train: nan (+/- nan)

================================================================================

================================================================================
✓ FILE 3/7 COMPLETED SUCCESSFULLY
================================================================================

✓ [3/7] Monday-WorkingHours.pcap_ISCX.csv - SUCCESS

################################################################################
# PROCESSING FILE 4/7: Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================
✓ Data loaded successfully!
  Rows: 288,602
  Columns: 85
✓ Column names cleaned (whitespace stripped)

================================================================================
STEP 2: CREATING BINARY LABELS
================================================================================
================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  BENIGN: 288,566 (99.99%)
  Infiltration: 36 (0.01%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0
  'Infiltration' → 1

✓ Binary labels created in 'Binary_Label' column
================================================================================

================================================================================
STEP 3: UNSUPERVISED LEARNING (DBSCAN)
================================================================================
================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 288,602

2. Handling infinite values...
   Replaced 396 infinite values with NaN

3. Handling missing values...
   Imputed 414 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 25 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Fwd IAT Total', 'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Min', 'Fwd Header Length', 'Bwd Header Length', 'Fwd Packets/s']

5. Features after correlation removal: 55

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 288,602 samples × 55 features
================================================================================
================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 55

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 88.23%

✓ PCA complete!
================================================================================
================================================================================
APPLYING DBSCAN CLUSTERING
================================================================================

Parameters:
  eps = 0.5
  min_samples = 10

Clustering 288,602 samples with 20 features...

✓ Clustering complete in 22.40 seconds

Results:
  Clusters found: 582
  Noise points: 26,624 (9.2%)
  Clustered points: 261,978 (90.8%)

Cluster size distribution:
  Cluster 0: 2,860 points (1.0%)
  Cluster 1: 771 points (0.3%)
  Cluster 2: 6,484 points (2.2%)
  Cluster 3: 47,597 points (16.5%)
  Cluster 4: 2,388 points (0.8%)
  Cluster 5: 11 points (0.0%)
  Cluster 6: 1,011 points (0.4%)
  Cluster 7: 39,223 points (13.6%)
  Cluster 8: 4,502 points (1.6%)
  Cluster 9: 1,059 points (0.4%)
  Cluster 10: 13,393 points (4.6%)
  Cluster 11: 462 points (0.2%)
  Cluster 12: 6,180 points (2.1%)
  Cluster 13: 230 points (0.1%)
  Cluster 14: 50 points (0.0%)
  Cluster 15: 75 points (0.0%)
  Cluster 16: 97 points (0.0%)
  Cluster 17: 1,112 points (0.4%)
  Cluster 18: 68 points (0.0%)
  Cluster 19: 1,254 points (0.4%)
  Cluster 20: 1,136 points (0.4%)
  Cluster 21: 130 points (0.0%)
  Cluster 22: 196 points (0.1%)
  Cluster 23: 995 points (0.3%)
  Cluster 24: 7,263 points (2.5%)
  Cluster 25: 1,714 points (0.6%)
  Cluster 26: 33 points (0.0%)
  Cluster 27: 23 points (0.0%)
  Cluster 28: 257 points (0.1%)
  Cluster 29: 153 points (0.1%)
  Cluster 30: 237 points (0.1%)
  Cluster 31: 212 points (0.1%)
  Cluster 32: 203 points (0.1%)
  Cluster 33: 976 points (0.3%)
  Cluster 34: 129 points (0.0%)
  Cluster 35: 996 points (0.3%)
  Cluster 36: 87 points (0.0%)
  Cluster 37: 36 points (0.0%)
  Cluster 38: 45 points (0.0%)
  Cluster 39: 102 points (0.0%)
  Cluster 40: 1,296 points (0.4%)
  Cluster 41: 52 points (0.0%)
  Cluster 42: 14 points (0.0%)
  Cluster 43: 44 points (0.0%)
  Cluster 44: 310 points (0.1%)
  Cluster 45: 58 points (0.0%)
  Cluster 46: 79 points (0.0%)
  Cluster 47: 129 points (0.0%)
  Cluster 48: 337 points (0.1%)
  Cluster 49: 350 points (0.1%)
  Cluster 50: 12 points (0.0%)
  Cluster 51: 264 points (0.1%)
  Cluster 52: 46 points (0.0%)
  Cluster 53: 85 points (0.0%)
  Cluster 54: 143 points (0.0%)
  Cluster 55: 24 points (0.0%)
  Cluster 56: 27 points (0.0%)
  Cluster 57: 19 points (0.0%)
  Cluster 58: 14 points (0.0%)
  Cluster 59: 147 points (0.1%)
  Cluster 60: 16 points (0.0%)
  Cluster 61: 119 points (0.0%)
  Cluster 62: 69 points (0.0%)
  Cluster 63: 3,154 points (1.1%)
  Cluster 64: 124 points (0.0%)
  Cluster 65: 52 points (0.0%)
  Cluster 66: 193 points (0.1%)
  Cluster 67: 109 points (0.0%)
  Cluster 68: 55 points (0.0%)
  Cluster 69: 387 points (0.1%)
  Cluster 70: 28 points (0.0%)
  Cluster 71: 271 points (0.1%)
  Cluster 72: 14 points (0.0%)
  Cluster 73: 24 points (0.0%)
  Cluster 74: 737 points (0.3%)
  Cluster 75: 7,308 points (2.5%)
  Cluster 76: 865 points (0.3%)
  Cluster 77: 127 points (0.0%)
  Cluster 78: 929 points (0.3%)
  Cluster 79: 299 points (0.1%)
  Cluster 80: 64 points (0.0%)
  Cluster 81: 240 points (0.1%)
  Cluster 82: 95 points (0.0%)
  Cluster 83: 99 points (0.0%)
  Cluster 84: 44 points (0.0%)
  Cluster 85: 140 points (0.0%)
  Cluster 86: 149 points (0.1%)
  Cluster 87: 305 points (0.1%)
  Cluster 88: 1,725 points (0.6%)
  Cluster 89: 2,357 points (0.8%)
  Cluster 90: 161 points (0.1%)
  Cluster 91: 207 points (0.1%)
  Cluster 92: 25 points (0.0%)
  Cluster 93: 35 points (0.0%)
  Cluster 94: 14 points (0.0%)
  Cluster 95: 32 points (0.0%)
  Cluster 96: 13 points (0.0%)
  Cluster 97: 247 points (0.1%)
  Cluster 98: 21 points (0.0%)
  Cluster 99: 288 points (0.1%)
  Cluster 100: 15 points (0.0%)
  Cluster 101: 102 points (0.0%)
  Cluster 102: 87 points (0.0%)
  Cluster 103: 65 points (0.0%)
  Cluster 104: 633 points (0.2%)
  Cluster 105: 15 points (0.0%)
  Cluster 106: 441 points (0.2%)
  Cluster 107: 23 points (0.0%)
  Cluster 108: 1,937 points (0.7%)
  Cluster 109: 94 points (0.0%)
  Cluster 110: 114 points (0.0%)
  Cluster 111: 227 points (0.1%)
  Cluster 112: 21 points (0.0%)
  Cluster 113: 37 points (0.0%)
  Cluster 114: 225 points (0.1%)
  Cluster 115: 435 points (0.2%)
  Cluster 116: 395 points (0.1%)
  Cluster 117: 109 points (0.0%)
  Cluster 118: 54 points (0.0%)
  Cluster 119: 31 points (0.0%)
  Cluster 120: 20 points (0.0%)
  Cluster 121: 71 points (0.0%)
  Cluster 122: 21 points (0.0%)
  Cluster 123: 36 points (0.0%)
  Cluster 124: 770 points (0.3%)
  Cluster 125: 81 points (0.0%)
  Cluster 126: 52 points (0.0%)
  Cluster 127: 46 points (0.0%)
  Cluster 128: 20 points (0.0%)
  Cluster 129: 19 points (0.0%)
  Cluster 130: 10 points (0.0%)
  Cluster 131: 10 points (0.0%)
  Cluster 132: 183 points (0.1%)
  Cluster 133: 104 points (0.0%)
  Cluster 134: 29 points (0.0%)
  Cluster 135: 31 points (0.0%)
  Cluster 136: 128 points (0.0%)
  Cluster 137: 39 points (0.0%)
  Cluster 138: 43 points (0.0%)
  Cluster 139: 93 points (0.0%)
  Cluster 140: 18 points (0.0%)
  Cluster 141: 54 points (0.0%)
  Cluster 142: 32 points (0.0%)
  Cluster 143: 19 points (0.0%)
  Cluster 144: 13 points (0.0%)
  Cluster 145: 160 points (0.1%)
  Cluster 146: 82 points (0.0%)
  Cluster 147: 162 points (0.1%)
  Cluster 148: 26 points (0.0%)
  Cluster 149: 227 points (0.1%)
  Cluster 150: 19 points (0.0%)
  Cluster 151: 17 points (0.0%)
  Cluster 152: 19 points (0.0%)
  Cluster 153: 13 points (0.0%)
  Cluster 154: 16 points (0.0%)
  Cluster 155: 15 points (0.0%)
  Cluster 156: 704 points (0.2%)
  Cluster 157: 57 points (0.0%)
  Cluster 158: 251 points (0.1%)
  Cluster 159: 43 points (0.0%)
  Cluster 160: 21 points (0.0%)
  Cluster 161: 43 points (0.0%)
  Cluster 162: 266 points (0.1%)
  Cluster 163: 281 points (0.1%)
  Cluster 164: 103 points (0.0%)
  Cluster 165: 19 points (0.0%)
  Cluster 166: 23 points (0.0%)
  Cluster 167: 40 points (0.0%)
  Cluster 168: 27 points (0.0%)
  Cluster 169: 17 points (0.0%)
  Cluster 170: 26 points (0.0%)
  Cluster 171: 77 points (0.0%)
  Cluster 172: 16 points (0.0%)
  Cluster 173: 19 points (0.0%)
  Cluster 174: 823 points (0.3%)
  Cluster 175: 10 points (0.0%)
  Cluster 176: 31 points (0.0%)
  Cluster 177: 82 points (0.0%)
  Cluster 178: 12 points (0.0%)
  Cluster 179: 35 points (0.0%)
  Cluster 180: 21 points (0.0%)
  Cluster 181: 32 points (0.0%)
  Cluster 182: 254 points (0.1%)
  Cluster 183: 317 points (0.1%)
  Cluster 184: 245 points (0.1%)
  Cluster 185: 119 points (0.0%)
  Cluster 186: 23 points (0.0%)
  Cluster 187: 19 points (0.0%)
  Cluster 188: 31 points (0.0%)
  Cluster 189: 15 points (0.0%)
  Cluster 190: 203 points (0.1%)
  Cluster 191: 71 points (0.0%)
  Cluster 192: 266 points (0.1%)
  Cluster 193: 359 points (0.1%)
  Cluster 194: 39 points (0.0%)
  Cluster 195: 30 points (0.0%)
  Cluster 196: 663 points (0.2%)
  Cluster 197: 29 points (0.0%)
  Cluster 198: 12 points (0.0%)
  Cluster 199: 11 points (0.0%)
  Cluster 200: 19 points (0.0%)
  Cluster 201: 31 points (0.0%)
  Cluster 202: 53 points (0.0%)
  Cluster 203: 1,680 points (0.6%)
  Cluster 204: 81 points (0.0%)
  Cluster 205: 48 points (0.0%)
  Cluster 206: 16 points (0.0%)
  Cluster 207: 27 points (0.0%)
  Cluster 208: 113 points (0.0%)
  Cluster 209: 15 points (0.0%)
  Cluster 210: 95 points (0.0%)
  Cluster 211: 132 points (0.0%)
  Cluster 212: 26 points (0.0%)
  Cluster 213: 16 points (0.0%)
  Cluster 214: 17 points (0.0%)
  Cluster 215: 37 points (0.0%)
  Cluster 216: 52 points (0.0%)
  Cluster 217: 20 points (0.0%)
  Cluster 218: 10 points (0.0%)
  Cluster 219: 34 points (0.0%)
  Cluster 220: 41 points (0.0%)
  Cluster 221: 28 points (0.0%)
  Cluster 222: 17 points (0.0%)
  Cluster 223: 21 points (0.0%)
  Cluster 224: 43 points (0.0%)
  Cluster 225: 21 points (0.0%)
  Cluster 226: 43 points (0.0%)
  Cluster 227: 19 points (0.0%)
  Cluster 228: 115 points (0.0%)
  Cluster 229: 402 points (0.1%)
  Cluster 230: 118 points (0.0%)
  Cluster 231: 20 points (0.0%)
  Cluster 232: 15 points (0.0%)
  Cluster 233: 40 points (0.0%)
  Cluster 234: 19 points (0.0%)
  Cluster 235: 32 points (0.0%)
  Cluster 236: 18 points (0.0%)
  Cluster 237: 70 points (0.0%)
  Cluster 238: 109 points (0.0%)
  Cluster 239: 51 points (0.0%)
  Cluster 240: 66 points (0.0%)
  Cluster 241: 13 points (0.0%)
  Cluster 242: 13 points (0.0%)
  Cluster 243: 13 points (0.0%)
  Cluster 244: 36 points (0.0%)
  Cluster 245: 10 points (0.0%)
  Cluster 246: 11 points (0.0%)
  Cluster 247: 15 points (0.0%)
  Cluster 248: 10 points (0.0%)
  Cluster 249: 38 points (0.0%)
  Cluster 250: 39 points (0.0%)
  Cluster 251: 82 points (0.0%)
  Cluster 252: 13 points (0.0%)
  Cluster 253: 102 points (0.0%)
  Cluster 254: 38 points (0.0%)
  Cluster 255: 67 points (0.0%)
  Cluster 256: 67 points (0.0%)
  Cluster 257: 37 points (0.0%)
  Cluster 258: 30 points (0.0%)
  Cluster 259: 17 points (0.0%)
  Cluster 260: 140 points (0.0%)
  Cluster 261: 23 points (0.0%)
  Cluster 262: 11 points (0.0%)
  Cluster 263: 18 points (0.0%)
  Cluster 264: 126 points (0.0%)
  Cluster 265: 16 points (0.0%)
  Cluster 266: 13 points (0.0%)
  Cluster 267: 10 points (0.0%)
  Cluster 268: 26 points (0.0%)
  Cluster 269: 22 points (0.0%)
  Cluster 270: 55 points (0.0%)
  Cluster 271: 12 points (0.0%)
  Cluster 272: 23 points (0.0%)
  Cluster 273: 31 points (0.0%)
  Cluster 274: 38 points (0.0%)
  Cluster 275: 13 points (0.0%)
  Cluster 276: 10 points (0.0%)
  Cluster 277: 28 points (0.0%)
  Cluster 278: 42 points (0.0%)
  Cluster 279: 47 points (0.0%)
  Cluster 280: 72 points (0.0%)
  Cluster 281: 15 points (0.0%)
  Cluster 282: 51 points (0.0%)
  Cluster 283: 13 points (0.0%)
  Cluster 284: 23 points (0.0%)
  Cluster 285: 30 points (0.0%)
  Cluster 286: 14 points (0.0%)
  Cluster 287: 15 points (0.0%)
  Cluster 288: 13 points (0.0%)
  Cluster 289: 16 points (0.0%)
  Cluster 290: 12 points (0.0%)
  Cluster 291: 30 points (0.0%)
  Cluster 292: 23 points (0.0%)
  Cluster 293: 50 points (0.0%)
  Cluster 294: 92 points (0.0%)
  Cluster 295: 17 points (0.0%)
  Cluster 296: 34 points (0.0%)
  Cluster 297: 44 points (0.0%)
  Cluster 298: 19 points (0.0%)
  Cluster 299: 10 points (0.0%)
  Cluster 300: 34 points (0.0%)
  Cluster 301: 14 points (0.0%)
  Cluster 302: 10 points (0.0%)
  Cluster 303: 13 points (0.0%)
  Cluster 304: 15 points (0.0%)
  Cluster 305: 12 points (0.0%)
  Cluster 306: 12 points (0.0%)
  Cluster 307: 12 points (0.0%)
  Cluster 308: 10 points (0.0%)
  Cluster 309: 15 points (0.0%)
  Cluster 310: 37 points (0.0%)
  Cluster 311: 39 points (0.0%)
  Cluster 312: 83 points (0.0%)
  Cluster 313: 17 points (0.0%)
  Cluster 314: 25 points (0.0%)
  Cluster 315: 15 points (0.0%)
  Cluster 316: 15 points (0.0%)
  Cluster 317: 13 points (0.0%)
  Cluster 318: 20 points (0.0%)
  Cluster 319: 23 points (0.0%)
  Cluster 320: 16 points (0.0%)
  Cluster 321: 14 points (0.0%)
  Cluster 322: 31 points (0.0%)
  Cluster 323: 32 points (0.0%)
  Cluster 324: 46 points (0.0%)
  Cluster 325: 15 points (0.0%)
  Cluster 326: 14 points (0.0%)
  Cluster 327: 70 points (0.0%)
  Cluster 328: 15 points (0.0%)
  Cluster 329: 72 points (0.0%)
  Cluster 330: 65 points (0.0%)
  Cluster 331: 57 points (0.0%)
  Cluster 332: 139 points (0.0%)
  Cluster 333: 27 points (0.0%)
  Cluster 334: 37 points (0.0%)
  Cluster 335: 50 points (0.0%)
  Cluster 336: 212 points (0.1%)
  Cluster 337: 38 points (0.0%)
  Cluster 338: 15 points (0.0%)
  Cluster 339: 56 points (0.0%)
  Cluster 340: 250 points (0.1%)
  Cluster 341: 128 points (0.0%)
  Cluster 342: 207 points (0.1%)
  Cluster 343: 25 points (0.0%)
  Cluster 344: 18 points (0.0%)
  Cluster 345: 90 points (0.0%)
  Cluster 346: 15 points (0.0%)
  Cluster 347: 13 points (0.0%)
  Cluster 348: 14 points (0.0%)
  Cluster 349: 20 points (0.0%)
  Cluster 350: 11 points (0.0%)
  Cluster 351: 20 points (0.0%)
  Cluster 352: 11 points (0.0%)
  Cluster 353: 11 points (0.0%)
  Cluster 354: 22 points (0.0%)
  Cluster 355: 10 points (0.0%)
  Cluster 356: 41 points (0.0%)
  Cluster 357: 11 points (0.0%)
  Cluster 358: 11 points (0.0%)
  Cluster 359: 31 points (0.0%)
  Cluster 360: 121 points (0.0%)
  Cluster 361: 11 points (0.0%)
  Cluster 362: 275 points (0.1%)
  Cluster 363: 25 points (0.0%)
  Cluster 364: 46 points (0.0%)
  Cluster 365: 65 points (0.0%)
  Cluster 366: 16 points (0.0%)
  Cluster 367: 13 points (0.0%)
  Cluster 368: 16 points (0.0%)
  Cluster 369: 22 points (0.0%)
  Cluster 370: 11 points (0.0%)
  Cluster 371: 8 points (0.0%)
  Cluster 372: 16 points (0.0%)
  Cluster 373: 23 points (0.0%)
  Cluster 374: 186 points (0.1%)
  Cluster 375: 125 points (0.0%)
  Cluster 376: 21 points (0.0%)
  Cluster 377: 29 points (0.0%)
  Cluster 378: 31,622 points (11.0%)
  Cluster 379: 17 points (0.0%)
  Cluster 380: 24 points (0.0%)
  Cluster 381: 51 points (0.0%)
  Cluster 382: 31 points (0.0%)
  Cluster 383: 42 points (0.0%)
  Cluster 384: 12 points (0.0%)
  Cluster 385: 11 points (0.0%)
  Cluster 386: 15 points (0.0%)
  Cluster 387: 30 points (0.0%)
  Cluster 388: 25 points (0.0%)
  Cluster 389: 18 points (0.0%)
  Cluster 390: 14 points (0.0%)
  Cluster 391: 14 points (0.0%)
  Cluster 392: 20 points (0.0%)
  Cluster 393: 11 points (0.0%)
  Cluster 394: 20 points (0.0%)
  Cluster 395: 13 points (0.0%)
  Cluster 396: 10 points (0.0%)
  Cluster 397: 142 points (0.0%)
  Cluster 398: 40 points (0.0%)
  Cluster 399: 14 points (0.0%)
  Cluster 400: 10 points (0.0%)
  Cluster 401: 14 points (0.0%)
  Cluster 402: 13 points (0.0%)
  Cluster 403: 9 points (0.0%)
  Cluster 404: 21 points (0.0%)
  Cluster 405: 28 points (0.0%)
  Cluster 406: 20 points (0.0%)
  Cluster 407: 34 points (0.0%)
  Cluster 408: 24 points (0.0%)
  Cluster 409: 32 points (0.0%)
  Cluster 410: 10 points (0.0%)
  Cluster 411: 10 points (0.0%)
  Cluster 412: 30 points (0.0%)
  Cluster 413: 12 points (0.0%)
  Cluster 414: 43 points (0.0%)
  Cluster 415: 13 points (0.0%)
  Cluster 416: 14 points (0.0%)
  Cluster 417: 76 points (0.0%)
  Cluster 418: 11 points (0.0%)
  Cluster 419: 11 points (0.0%)
  Cluster 420: 27 points (0.0%)
  Cluster 421: 13 points (0.0%)
  Cluster 422: 10 points (0.0%)
  Cluster 423: 12 points (0.0%)
  Cluster 424: 25 points (0.0%)
  Cluster 425: 14 points (0.0%)
  Cluster 426: 66 points (0.0%)
  Cluster 427: 14 points (0.0%)
  Cluster 428: 15 points (0.0%)
  Cluster 429: 15 points (0.0%)
  Cluster 430: 21 points (0.0%)
  Cluster 431: 12 points (0.0%)
  Cluster 432: 47 points (0.0%)
  Cluster 433: 10 points (0.0%)
  Cluster 434: 11 points (0.0%)
  Cluster 435: 31 points (0.0%)
  Cluster 436: 20 points (0.0%)
  Cluster 437: 435 points (0.2%)
  Cluster 438: 28,093 points (9.7%)
  Cluster 439: 89 points (0.0%)
  Cluster 440: 577 points (0.2%)
  Cluster 441: 2,178 points (0.8%)
  Cluster 442: 12 points (0.0%)
  Cluster 443: 13 points (0.0%)
  Cluster 444: 44 points (0.0%)
  Cluster 445: 22 points (0.0%)
  Cluster 446: 136 points (0.0%)
  Cluster 447: 120 points (0.0%)
  Cluster 448: 11 points (0.0%)
  Cluster 449: 34 points (0.0%)
  Cluster 450: 14 points (0.0%)
  Cluster 451: 22 points (0.0%)
  Cluster 452: 22 points (0.0%)
  Cluster 453: 10 points (0.0%)
  Cluster 454: 19 points (0.0%)
  Cluster 455: 47 points (0.0%)
  Cluster 456: 10 points (0.0%)
  Cluster 457: 14 points (0.0%)
  Cluster 458: 10 points (0.0%)
  Cluster 459: 40 points (0.0%)
  Cluster 460: 106 points (0.0%)
  Cluster 461: 16 points (0.0%)
  Cluster 462: 14 points (0.0%)
  Cluster 463: 29 points (0.0%)
  Cluster 464: 40 points (0.0%)
  Cluster 465: 12 points (0.0%)
  Cluster 466: 59 points (0.0%)
  Cluster 467: 16 points (0.0%)
  Cluster 468: 83 points (0.0%)
  Cluster 469: 18 points (0.0%)
  Cluster 470: 12 points (0.0%)
  Cluster 471: 11 points (0.0%)
  Cluster 472: 22 points (0.0%)
  Cluster 473: 15 points (0.0%)
  Cluster 474: 1,484 points (0.5%)
  Cluster 475: 1,115 points (0.4%)
  Cluster 476: 10 points (0.0%)
  Cluster 477: 15 points (0.0%)
  Cluster 478: 98 points (0.0%)
  Cluster 479: 30 points (0.0%)
  Cluster 480: 54 points (0.0%)
  Cluster 481: 13 points (0.0%)
  Cluster 482: 38 points (0.0%)
  Cluster 483: 10 points (0.0%)
  Cluster 484: 11 points (0.0%)
  Cluster 485: 25 points (0.0%)
  Cluster 486: 9 points (0.0%)
  Cluster 487: 11 points (0.0%)
  Cluster 488: 63 points (0.0%)
  Cluster 489: 32 points (0.0%)
  Cluster 490: 22 points (0.0%)
  Cluster 491: 29 points (0.0%)
  Cluster 492: 10 points (0.0%)
  Cluster 493: 18 points (0.0%)
  Cluster 494: 159 points (0.1%)
  Cluster 495: 121 points (0.0%)
  Cluster 496: 158 points (0.1%)
  Cluster 497: 10 points (0.0%)
  Cluster 498: 24 points (0.0%)
  Cluster 499: 15 points (0.0%)
  Cluster 500: 24 points (0.0%)
  Cluster 501: 11 points (0.0%)
  Cluster 502: 37 points (0.0%)
  Cluster 503: 37 points (0.0%)
  Cluster 504: 19 points (0.0%)
  Cluster 505: 22 points (0.0%)
  Cluster 506: 22 points (0.0%)
  Cluster 507: 37 points (0.0%)
  Cluster 508: 19 points (0.0%)
  Cluster 509: 50 points (0.0%)
  Cluster 510: 18 points (0.0%)
  Cluster 511: 80 points (0.0%)
  Cluster 512: 35 points (0.0%)
  Cluster 513: 20 points (0.0%)
  Cluster 514: 20 points (0.0%)
  Cluster 515: 72 points (0.0%)
  Cluster 516: 111 points (0.0%)
  Cluster 517: 84 points (0.0%)
  Cluster 518: 10 points (0.0%)
  Cluster 519: 10 points (0.0%)
  Cluster 520: 32 points (0.0%)
  Cluster 521: 16 points (0.0%)
  Cluster 522: 12 points (0.0%)
  Cluster 523: 19 points (0.0%)
  Cluster 524: 11 points (0.0%)
  Cluster 525: 11 points (0.0%)
  Cluster 526: 17 points (0.0%)
  Cluster 527: 14 points (0.0%)
  Cluster 528: 25 points (0.0%)
  Cluster 529: 11 points (0.0%)
  Cluster 530: 19 points (0.0%)
  Cluster 531: 14 points (0.0%)
  Cluster 532: 11 points (0.0%)
  Cluster 533: 23 points (0.0%)
  Cluster 534: 10 points (0.0%)
  Cluster 535: 8 points (0.0%)
  Cluster 536: 15 points (0.0%)
  Cluster 537: 10 points (0.0%)
  Cluster 538: 10 points (0.0%)
  Cluster 539: 12 points (0.0%)
  Cluster 540: 10 points (0.0%)
  Cluster 541: 14 points (0.0%)
  Cluster 542: 10 points (0.0%)
  Cluster 543: 10 points (0.0%)
  Cluster 544: 13 points (0.0%)
  Cluster 545: 10 points (0.0%)
  Cluster 546: 10 points (0.0%)
  Cluster 547: 10 points (0.0%)
  Cluster 548: 18 points (0.0%)
  Cluster 549: 9 points (0.0%)
  Cluster 550: 11 points (0.0%)
  Cluster 551: 10 points (0.0%)
  Cluster 552: 33 points (0.0%)
  Cluster 553: 12 points (0.0%)
  Cluster 554: 12 points (0.0%)
  Cluster 555: 49 points (0.0%)
  Cluster 556: 13 points (0.0%)
  Cluster 557: 280 points (0.1%)
  Cluster 558: 13 points (0.0%)
  Cluster 559: 11 points (0.0%)
  Cluster 560: 13 points (0.0%)
  Cluster 561: 11 points (0.0%)
  Cluster 562: 15 points (0.0%)
  Cluster 563: 10 points (0.0%)
  Cluster 564: 16 points (0.0%)
  Cluster 565: 16 points (0.0%)
  Cluster 566: 13 points (0.0%)
  Cluster 567: 10 points (0.0%)
  Cluster 568: 10 points (0.0%)
  Cluster 569: 10 points (0.0%)
  Cluster 570: 11 points (0.0%)
  Cluster 571: 9 points (0.0%)
  Cluster 572: 10 points (0.0%)
  Cluster 573: 10 points (0.0%)
  Cluster 574: 11 points (0.0%)
  Cluster 575: 10 points (0.0%)
  Cluster 576: 10 points (0.0%)
  Cluster 577: 12 points (0.0%)
  Cluster 578: 41 points (0.0%)
  Cluster 579: 10 points (0.0%)
  Cluster 580: 8 points (0.0%)
  Cluster 581: 8 points (0.0%)

Generating visualization...

================================================================================

================================================================================
STEP 4: SUPERVISED LEARNING (XGBOOST)
================================================================================
================================================================================
FEATURE PREPARATION FOR MACHINE LEARNING
================================================================================

1. Removing 8 non-feature columns:
   - Flow ID
   - Source IP
   - Destination IP
   - Timestamp
   - Label
   - Binary_Label
   - Source Port
   - Destination Port

2. Ensuring all features are numeric...
   ✓ All features are numeric

3. Handling infinite values...
   Found 396 infinite values
   ✓ Replaced with NaN

4. Handling missing values...
   Found 414 missing values
   Imputing with column medians...
   ✓ Missing values imputed

================================================================================
FINAL DATASET FOR MODELING
================================================================================

Samples: 288,602
Features: 78

Class distribution:
  Class 0 (BENIGN): 288,566 (99.99%)
  Class 1 (ATTACK): 36 (0.01%)

Feature types:
  Port features: 0
  Other features: 78

Sample feature names:
  1. Protocol
  2. Flow Duration
  3. Total Fwd Packets
  4. Total Backward Packets
  5. Total Length of Fwd Packets
  6. Total Length of Bwd Packets
  7. Fwd Packet Length Max
  8. Fwd Packet Length Min
  9. Fwd Packet Length Mean
  10. Fwd Packet Length Std
  ... and 68 more

================================================================================
================================================================================
MODEL 3: XGBOOST
================================================================================

📊 Training with 10-fold cross-validation...
   This may take several minutes...

Calculated scale_pos_weight: 8015.72
  (Ratio of negative to positive samples)

Creating XGBoost model...
Performing 10-fold cross-validation...

✓ Cross-validation complete in 5.66 seconds

================================================================================
XGBOOST - 10-FOLD CV RESULTS
================================================================================

ACCURACY:
  Test:  1.0000 (+/- 0.0000)
  Train: 1.0000 (+/- 0.0000)

PRECISION:
  Test:  0.9083 (+/- 0.1417)
  Train: 0.9685 (+/- 0.0346)

RECALL:
  Test:  0.7833 (+/- 0.1944)
  Train: 1.0000 (+/- 0.0000)

F1:
  Test:  0.8171 (+/- 0.1184)
  Train: 0.9837 (+/- 0.0180)

ROC_AUC:
  Test:  0.9413 (+/- 0.0991)
  Train: 1.0000 (+/- 0.0000)

================================================================================

================================================================================
✓ FILE 4/7 COMPLETED SUCCESSFULLY
================================================================================

✓ [4/7] Thursday-WorkingHours-Afternoon-Infilteration.pcap_ISCX.csv - SUCCESS

################################################################################
# PROCESSING FILE 5/7: Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================
✗ Error loading data: 'utf-8' codec can't decode byte 0x96 in position 22398: invalid start byte

❌ [5/7] Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv - FAILED: Failed to load data

################################################################################
# PROCESSING FILE 6/7: Tuesday-WorkingHours.pcap_ISCX.csv
################################################################################

================================================================================
STEP 1: LOADING DATA
================================================================================
✓ Data loaded successfully!
  Rows: 445,909
  Columns: 85
✓ Column names cleaned (whitespace stripped)

================================================================================
STEP 2: CREATING BINARY LABELS
================================================================================
================================================================================
CREATING BINARY LABELS
================================================================================

Original labels in 'Label' column:
  BENIGN: 432,074 (96.90%)
  FTP-Patator: 7,938 (1.78%)
  SSH-Patator: 5,897 (1.32%)

--------------------------------------------------------------------------------
BINARY ENCODING
--------------------------------------------------------------------------------
Class 0 (BENIGN): 'BENIGN'
Class 1 (ATTACK): Everything else

Mapping examples:
  'BENIGN' → 0
  'FTP-Patator' → 1
  'SSH-Patator' → 1

✓ Binary labels created in 'Binary_Label' column
================================================================================

================================================================================
STEP 3: UNSUPERVISED LEARNING (DBSCAN)
================================================================================
================================================================================
PREPROCESSING FOR CLUSTERING
================================================================================

1. Starting features: 80
   Starting samples: 445,909

2. Handling infinite values...
   Replaced 327 infinite values with NaN

3. Handling missing values...
   Imputed 528 missing values with column medians

4. Removing highly correlated features (threshold=0.95)...
   Removing 22 highly correlated features
   First 10 removed: ['Total Backward Packets', 'Total Length of Bwd Packets', 'Fwd IAT Total', 'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Min', 'Fwd Packets/s', 'Packet Length Std', 'SYN Flag Count']

5. Features after correlation removal: 58

6. Scaling features...

✓ Preprocessing complete!
  Final shape: 445,909 samples × 58 features
================================================================================
================================================================================
DIMENSIONALITY REDUCTION WITH PCA
================================================================================

Original dimensions: 58

Capping at 20 components for computational efficiency

Reduced to: 20 components
Variance explained: 89.11%

def create_multi_file_comparison(all_results):
    """
    Create comparison table across all processed files.
    """
    print("=" * 80)
    print("MULTI-FILE COMPARISON")
    print("=" * 80)
    
    # Filter successful results
    successful_results = [r for r in all_results if r['success']]
    
    if not successful_results:
        print("\n⚠️ No successful results to compare")
        return None
    
    # Create comparison data
    comparison_data = []
    
    for result in successful_results:
        row = {
            'File': result['file_path'],
            'Samples': result['n_samples'],
            'Features': result['n_features'],
            'Benign': result['class_distribution']['benign'],
            'Attack': result['class_distribution']['attack'],
            'Imbalance': f"{result['class_distribution']['imbalance_ratio']:.2f}",
            'DBSCAN_Clusters': result['dbscan']['n_clusters'],
            'DBSCAN_Noise%': f"{result['dbscan']['noise_percentage']:.1f}%",
            'XGB_Accuracy': f"{result['xgboost']['accuracy']:.4f}",
            'XGB_Precision': f"{result['xgboost']['precision']:.4f}",
            'XGB_Recall': f"{result['xgboost']['recall']:.4f}",
            'XGB_F1': f"{result['xgboost']['f1']:.4f}",
            'XGB_ROC_AUC': f"{result['xgboost']['roc_auc']:.4f}"
        }
        comparison_data.append(row)
    
    comparison_df = pd.DataFrame(comparison_data)
    
    print("\n📊 DATASET CHARACTERISTICS:")
    print("=" * 80)
    display(comparison_df[['File', 'Samples', 'Features', 'Benign', 'Attack', 'Imbalance']])
    
    print("\n🔍 DBSCAN CLUSTERING RESULTS:")
    print("=" * 80)
    display(comparison_df[['File', 'DBSCAN_Clusters', 'DBSCAN_Noise%']])
    
    print("\n🎯 XGBOOST CLASSIFICATION RESULTS (10-FOLD CV):")
    print("=" * 80)
    display(comparison_df[['File', 'XGB_Accuracy', 'XGB_Precision', 'XGB_Recall', 'XGB_F1', 'XGB_ROC_AUC']])
    
    # Calculate averages
    print("\n" + "=" * 80)
    print("AVERAGE PERFORMANCE ACROSS ALL FILES")
    print("=" * 80)
    
    avg_accuracy = np.mean([r['xgboost']['accuracy'] for r in successful_results])
    avg_precision = np.mean([r['xgboost']['precision'] for r in successful_results])
    avg_recall = np.mean([r['xgboost']['recall'] for r in successful_results])
    avg_f1 = np.mean([r['xgboost']['f1'] for r in successful_results])
    avg_roc_auc = np.mean([r['xgboost']['roc_auc'] for r in successful_results])
    
    print(f"\nXGBoost Average Performance:")
    print(f"  Accuracy:  {avg_accuracy:.4f}")
    print(f"  Precision: {avg_precision:.4f}")
    print(f"  Recall:    {avg_recall:.4f}")
    print(f"  F1-Score:  {avg_f1:.4f}")
    print(f"  ROC-AUC:   {avg_roc_auc:.4f}")
    
    print("\n" + "=" * 80)
    
    return comparison_df

# Create comparison
if 'all_results' in locals():
    comparison_df_multi = create_multi_file_comparison(all_results)
else:
    print("⚠️ Please run batch processing first")

================================================================================
MULTI-FILE COMPARISON
================================================================================

📊 DATASET CHARACTERISTICS:
================================================================================

🔍 DBSCAN CLUSTERING RESULTS:
================================================================================

🎯 XGBOOST CLASSIFICATION RESULTS (10-FOLD CV):
================================================================================

================================================================================
AVERAGE PERFORMANCE ACROSS ALL FILES
================================================================================

XGBoost Average Performance:
  Accuracy:  0.9998
  Precision: 0.8097
  Recall:    0.7961
  F1-Score:  0.7980
  ROC-AUC:   nan

================================================================================

def visualize_multi_file_results(all_results):
    """
    Create visualizations comparing results across files.
    """
    print("=" * 80)
    print("MULTI-FILE VISUALIZATION")
    print("=" * 80)
    
    # Filter successful results
    successful_results = [r for r in all_results if r['success']]
    
    if not successful_results:
        print("\n⚠️ No successful results to visualize")
        return
    
    fig, axes = plt.subplots(2, 3, figsize=(18, 10))
    axes = axes.flatten()
    
    file_names = [r['file_path'].split('/')[-1] for r in successful_results]
    
    # 1. Dataset sizes
    sizes = [r['n_samples'] for r in successful_results]
    axes[0].bar(range(len(sizes)), sizes, color='#3498db', alpha=0.7, edgecolor='black')
    axes[0].set_xlabel('File')
    axes[0].set_ylabel('Number of Samples')
    axes[0].set_title('Dataset Sizes', fontweight='bold')
    axes[0].set_xticks(range(len(file_names)))
    axes[0].set_xticklabels([f"F{i+1}" for i in range(len(file_names))], rotation=0)
    axes[0].grid(True, alpha=0.3, axis='y')
    
    # 2. Class imbalance ratios
    imbalance = [r['class_distribution']['imbalance_ratio'] for r in successful_results]
    axes[1].bar(range(len(imbalance)), imbalance, color='#e74c3c', alpha=0.7, edgecolor='black')
    axes[1].set_xlabel('File')
    axes[1].set_ylabel('Imbalance Ratio (Benign/Attack)')
    axes[1].set_title('Class Imbalance', fontweight='bold')
    axes[1].set_xticks(range(len(file_names)))
    axes[1].set_xticklabels([f"F{i+1}" for i in range(len(file_names))], rotation=0)
    axes[1].grid(True, alpha=0.3, axis='y')
    
    # 3. DBSCAN clusters
    clusters = [r['dbscan']['n_clusters'] for r in successful_results]
    axes[2].bar(range(len(clusters)), clusters, color='#2ecc71', alpha=0.7, edgecolor='black')
    axes[2].set_xlabel('File')
    axes[2].set_ylabel('Number of Clusters')
    axes[2].set_title('DBSCAN Clusters Found', fontweight='bold')
    axes[2].set_xticks(range(len(file_names)))
    axes[2].set_xticklabels([f"F{i+1}" for i in range(len(file_names))], rotation=0)
    axes[2].grid(True, alpha=0.3, axis='y')
    
    # 4. XGBoost F1-Score
    f1_scores = [r['xgboost']['f1'] for r in successful_results]
    f1_stds = [r['xgboost']['f1_std'] for r in successful_results]
    axes[3].bar(range(len(f1_scores)), f1_scores, yerr=f1_stds, 
               capsize=5, color='#f39c12', alpha=0.7, edgecolor='black')
    axes[3].set_xlabel('File')
    axes[3].set_ylabel('F1-Score')
    axes[3].set_title('XGBoost F1-Score (10-fold CV)', fontweight='bold')
    axes[3].set_xticks(range(len(file_names)))
    axes[3].set_xticklabels([f"F{i+1}" for i in range(len(file_names))], rotation=0)
    axes[3].set_ylim([0, 1.0])
    axes[3].grid(True, alpha=0.3, axis='y')
    
    # 5. XGBoost Precision vs Recall
    precisions = [r['xgboost']['precision'] for r in successful_results]
    recalls = [r['xgboost']['recall'] for r in successful_results]
    x = np.arange(len(file_names))
    width = 0.35
    axes[4].bar(x - width/2, precisions, width, label='Precision', color='#9b59b6', alpha=0.7, edgecolor='black')
    axes[4].bar(x + width/2, recalls, width, label='Recall', color='#1abc9c', alpha=0.7, edgecolor='black')
    axes[4].set_xlabel('File')
    axes[4].set_ylabel('Score')
    axes[4].set_title('XGBoost Precision vs Recall', fontweight='bold')
    axes[4].set_xticks(x)
    axes[4].set_xticklabels([f"F{i+1}" for i in range(len(file_names))], rotation=0)
    axes[4].set_ylim([0, 1.0])
    axes[4].legend()
    axes[4].grid(True, alpha=0.3, axis='y')
    
    # 6. Overall performance comparison
    metrics = ['Accuracy', 'Precision', 'Recall', 'F1', 'ROC-AUC']
    metric_keys = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
    avg_scores = [np.mean([r['xgboost'][key] for r in successful_results]) for key in metric_keys]
    
    axes[5].bar(range(len(metrics)), avg_scores, color='#34495e', alpha=0.7, edgecolor='black')
    axes[5].set_xlabel('Metric')
    axes[5].set_ylabel('Average Score')
    axes[5].set_title('Average XGBoost Performance', fontweight='bold')
    axes[5].set_xticks(range(len(metrics)))
    axes[5].set_xticklabels(metrics, rotation=45, ha='right')
    axes[5].set_ylim([0, 1.0])
    axes[5].grid(True, alpha=0.3, axis='y')
    
    # Add value labels on bars
    for i, score in enumerate(avg_scores):
        axes[5].text(i, score, f'{score:.3f}', ha='center', va='bottom', fontweight='bold')
    
    plt.suptitle(f'Multi-File Analysis - DBSCAN & XGBoost ({len(successful_results)} Files)', 
                fontsize=16, fontweight='bold', y=0.995)
    plt.tight_layout()
    plt.show()
    
    print("\n✓ Visualizations complete")

# Visualize results
if 'all_results' in locals():
    visualize_multi_file_results(all_results)
else:
    print("⚠️ Please run batch processing first")

================================================================================
MULTI-FILE VISUALIZATION
================================================================================

✓ Visualizations complete

# Uncomment to save:
# if 'comparison_df_multi' in locals():
#     comparison_df_multi.to_csv('multi_file_results.csv', index=False)
#     print("✓ Results saved to 'multi_file_results.csv'")

# Python version
Python 3.9+

# Required packages (install via pip or conda)
pip install pandas numpy scikit-learn xgboost matplotlib seaborn imbalanced-learn scipy

Model	Accuracy	Precision	Recall	F1-Score	ROC-AUC	Training Time*
XGBoost	97.3%	96.9%	96.7%	96.8%	99.2%	18 min
Random Forest	96.8%	96.2%	96.5%	96.3%	99.0%	6 min
Logistic Regression	92.1%	91.5%	91.8%	91.6%	97.1%	2 min

	Feature	Benign_Pct	Attack_Pct	Difference	Discrimination
6	dst_port_80_HTTP	9.135	99.998	90.862	90.862
16	dst_port_other	42.305	0.002	-42.303	42.303
5	src_port_other	58.313	99.998	41.685	41.685
0	src_port_80_HTTP	34.308	0.002	-34.306	34.306
10	dst_port_53_DNS	32.696	0.000	-32.696	32.696
7	dst_port_443_HTTPS	13.800	0.000	-13.800	13.800
1	src_port_443_HTTPS	6.569	0.000	-6.569	6.569
8	dst_port_8080_HTTP_Proxy	0.522	0.000	-0.522	0.522
3	src_port_123_NTP	0.370	0.000	-0.370	0.370
12	dst_port_123_NTP	0.370	0.000	-0.370	0.370
9	dst_port_22_SSH	0.350	0.000	-0.350	0.350
4	src_port_137_NetBIOS_Name	0.280	0.000	-0.280	0.280
13	dst_port_137_NetBIOS_Name	0.280	0.000	-0.280	0.280
15	dst_port_389_LDAP	0.267	0.000	-0.267	0.267
11	dst_port_21_FTP_Control	0.171	0.000	-0.171	0.171
2	src_port_22_SSH	0.160	0.000	-0.160	0.160
14	dst_port_139_NetBIOS_Session	0.102	0.000	-0.102	0.102

	Feature	Benign_Mean	Attack_Mean	Benign_Median	Attack_Median	Difference_Mean
0	Total Length of Fwd Packets	2128.513	31.909	70.0	26.0	-2096.604
1	Total Length of Bwd Packets	4109.004	7373.635	62.0	11601.0	3264.631
2	Fwd Packet Length Mean	371.081	7.401	38.0	7.0	-363.679
3	Bwd Packet Length Mean	116.896	1481.026	13.0	1934.5	1364.130

	Feature	Benign_Mean	Attack_Mean	Benign_Median	Attack_Median	Difference_Mean
0	Total Fwd Packets	5.402	4.472	2.000	4.000	-0.930
1	Total Backward Packets	6.298	3.256	2.000	4.000	-3.042
2	Flow Packets/s	32678.936	172.789	27.463	2.590	-32506.147
3	Fwd Packets/s	29009.143	102.137	14.426	1.673	-28907.006

	Feature	Benign_Mean_sec	Attack_Mean_sec	Benign_Median_sec	Attack_Median_sec	Difference_sec
0	Flow Duration	15.306	16.956	1.714e-01	1.879	1.649
1	Flow IAT Mean	1.186	1.882	5.964e-02	0.489	0.695
2	Fwd IAT Mean	2.391	2.655	4.000e-06	0.490	0.264
3	Bwd IAT Mean	2.043	0.112	4.900e-05	0.019	-1.931

	Flow ID	Source IP	Source Port	Destination IP	Destination Port	Protocol	Timestamp	Flow Duration	Total Fwd Packets	Total Backward Packets	Total Length of Fwd Packets	Total Length of Bwd Packets	Fwd Packet Length Max	Fwd Packet Length Min	Fwd Packet Length Mean	Bwd Packet Length Max	Bwd Packet Length Min	Bwd Packet Length Mean	Flow Bytes/s	Flow Packets/s	Flow IAT Mean	Flow IAT Max	Flow IAT Min	Fwd IAT Total	Fwd IAT Mean	Fwd IAT Max	Fwd IAT Min	Fwd Header Length	Bwd Header Length	Fwd Packets/s	Bwd Packets/s	Min Packet Length	Max Packet Length	Packet Length Mean	ACK Flag Count	URG Flag Count	Down/Up Ratio	Average Packet Size	Avg Fwd Segment Size	Avg Bwd Segment Size	Fwd Header Length.1	Subflow Fwd Packets	Subflow Fwd Bytes	Subflow Bwd Packets	Subflow Bwd Bytes	Init_Win_bytes_forward	Init_Win_bytes_backward	act_data_pkt_fwd	min_seg_size_forward	Label
0	192.168.10.5-104.16.207.165-54865-443-6	104.16.207.165	443	192.168.10.5	54865	6	7/7/2017 3:30	3	2	0	12	0	6	6	6.0	0	0	0.0	4.000e+06	666666.667	3.0	3	3	3	3.0	3	3	40	0	666666.667	0.000	6	6	6.0	1	0	0	9.0	6.0	0.0	40	2	12	0	0	33	-1	1	20	BENIGN
1	192.168.10.5-104.16.28.216-55054-80-6	104.16.28.216	80	192.168.10.5	55054	6	7/7/2017 3:30	109	1	1	6	6	6	6	6.0	6	6	6.0	1.101e+05	18348.624	109.0	109	109	0	0.0	0	0	20	20	9174.312	9174.312	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	29	256	0	20	BENIGN
2	192.168.10.5-104.16.28.216-55055-80-6	104.16.28.216	80	192.168.10.5	55055	6	7/7/2017 3:30	52	1	1	6	6	6	6	6.0	6	6	6.0	2.308e+05	38461.538	52.0	52	52	0	0.0	0	0	20	20	19230.769	19230.769	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	29	256	0	20	BENIGN
3	192.168.10.16-104.17.241.25-46236-443-6	104.17.241.25	443	192.168.10.16	46236	6	7/7/2017 3:30	34	1	1	6	6	6	6	6.0	6	6	6.0	3.529e+05	58823.529	34.0	34	34	0	0.0	0	0	20	20	29411.765	29411.765	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	31	329	0	20	BENIGN
4	192.168.10.5-104.19.196.102-54863-443-6	104.19.196.102	443	192.168.10.5	54863	6	7/7/2017 3:30	3	2	0	12	0	6	6	6.0	0	0	0.0	4.000e+06	666666.667	3.0	3	3	3	3.0	3	3	40	0	666666.667	0.000	6	6	6.0	1	0	0	9.0	6.0	0.0	40	2	12	0	0	32	-1	1	20	BENIGN

	Flow ID	Source IP	Source Port	Destination IP	Destination Port	Protocol	Timestamp	Flow Duration	Total Fwd Packets	Total Backward Packets	Total Length of Fwd Packets	Total Length of Bwd Packets	Fwd Packet Length Max	Fwd Packet Length Min	Fwd Packet Length Mean	Bwd Packet Length Max	Bwd Packet Length Min	Bwd Packet Length Mean	Flow Bytes/s	Flow Packets/s	Flow IAT Mean	Flow IAT Max	Flow IAT Min	Fwd IAT Total	Fwd IAT Mean	Fwd IAT Max	Fwd IAT Min	Fwd Header Length	Bwd Header Length	Fwd Packets/s	Bwd Packets/s	Min Packet Length	Max Packet Length	Packet Length Mean	ACK Flag Count	URG Flag Count	Down/Up Ratio	Average Packet Size	Avg Fwd Segment Size	Avg Bwd Segment Size	Fwd Header Length.1	Subflow Fwd Packets	Subflow Fwd Bytes	Subflow Bwd Packets	Subflow Bwd Bytes	Init_Win_bytes_forward	Init_Win_bytes_backward	act_data_pkt_fwd	min_seg_size_forward	Label
225740	192.168.10.15-72.21.91.29-61374-80-6	72.21.91.29	80	192.168.10.15	61374	6	7/7/2017 5:02	61	1	1	6	6	6	6	6.0	6	6	6.0	196721.312	32786.885	61.0	61	61	0	0.0	0	0	20	20	16393.443	16393.443	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	288	253	0	20	BENIGN
225741	192.168.10.15-72.21.91.29-61378-80-6	72.21.91.29	80	192.168.10.15	61378	6	7/7/2017 5:02	72	1	1	6	6	6	6	6.0	6	6	6.0	166666.667	27777.778	72.0	72	72	0	0.0	0	0	20	20	13888.889	13888.889	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	288	253	0	20	BENIGN
225742	192.168.10.15-72.21.91.29-61375-80-6	72.21.91.29	80	192.168.10.15	61375	6	7/7/2017 5:02	75	1	1	6	6	6	6	6.0	6	6	6.0	160000.000	26666.667	75.0	75	75	0	0.0	0	0	20	20	13333.333	13333.333	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	288	253	0	20	BENIGN
225743	192.168.10.15-8.41.222.187-61323-80-6	8.41.222.187	80	192.168.10.15	61323	6	7/7/2017 5:02	48	2	0	12	0	6	6	6.0	0	0	0.0	250000.000	41666.667	48.0	48	48	48	48.0	48	48	40	0	41666.667	0.000	6	6	6.0	1	0	0	9.0	6.0	0.0	40	2	12	0	0	4719	-1	1	20	BENIGN
225744	192.168.10.15-8.43.72.21-61326-80-6	8.43.72.21	80	192.168.10.15	61326	6	7/7/2017 5:02	68	1	1	6	6	6	6	6.0	6	6	6.0	176470.588	29411.765	68.0	68	68	0	0.0	0	0	20	20	14705.882	14705.882	6	6	6.0	1	1	1	9.0	6.0	6.0	20	1	6	1	6	13140	64240	0	20	BENIGN

	Source Port	Destination Port	Protocol	Flow Duration	Total Fwd Packets	Total Backward Packets	Total Length of Fwd Packets	Total Length of Bwd Packets	Fwd Packet Length Max	Fwd Packet Length Min	Fwd Packet Length Mean	Fwd Packet Length Std	Bwd Packet Length Max	Bwd Packet Length Min	Bwd Packet Length Mean	Bwd Packet Length Std	Flow Bytes/s	Flow Packets/s	Flow IAT Mean	Flow IAT Std	Flow IAT Max	Flow IAT Min	Fwd IAT Total	Fwd IAT Mean	Fwd IAT Std	Fwd IAT Max	Fwd IAT Min	Bwd IAT Total	Bwd IAT Mean	Bwd IAT Std	Bwd IAT Max	Bwd IAT Min	Fwd PSH Flags	Bwd PSH Flags	Fwd URG Flags	Bwd URG Flags	Fwd Header Length	Bwd Header Length	Fwd Packets/s	Bwd Packets/s	Min Packet Length	Max Packet Length	Packet Length Mean	Packet Length Std	Packet Length Variance	FIN Flag Count	SYN Flag Count	RST Flag Count	PSH Flag Count	ACK Flag Count	URG Flag Count	CWE Flag Count	ECE Flag Count	Down/Up Ratio	Average Packet Size	Avg Fwd Segment Size	Avg Bwd Segment Size	Fwd Header Length.1	Fwd Avg Bytes/Bulk	Fwd Avg Packets/Bulk	Fwd Avg Bulk Rate	Bwd Avg Bytes/Bulk	Bwd Avg Packets/Bulk	Bwd Avg Bulk Rate	Subflow Fwd Packets	Subflow Fwd Bytes	Subflow Bwd Packets	Subflow Bwd Bytes	Init_Win_bytes_forward	Init_Win_bytes_backward	act_data_pkt_fwd	min_seg_size_forward	Active Mean	Active Std	Active Max	Active Min	Idle Mean	Idle Std	Idle Max	Idle Min
count	225745.000	225745.000	225745.000	2.257e+05	225745.000	225745.000	225745.000	2.257e+05	225745.000	225745.000	225745.000	225745.000	225745.000	225745.000	225745.000	225745.000	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	225745.000	225745.0	225745.0	225745.0	225745.000	225745.000	2.257e+05	2.257e+05	225745.000	225745.000	225745.000	225745.000	2.257e+05	225745.000	225745.000	2.257e+05	225745.000	225745.000	225745.000	225745.0	2.257e+05	225745.000	225745.000	225745.000	225745.000	225745.000	225745.0	225745.0	225745.0	225745.0	225745.0	225745.0	225745.000	225745.000	225745.000	2.257e+05	225745.000	225745.000	225745.000	225745.000	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05	2.257e+05
mean	38257.568	8879.619	7.600	1.624e+07	4.875	4.573	939.463	5.960e+03	538.536	27.882	164.827	214.907	2735.585	16.719	890.537	1230.173	inf	inf	1.581e+06	4.249e+06	1.349e+07	2.812e+04	1.540e+07	2.541e+06	5.195e+06	1.299e+07	2.074e+05	6.565e+06	9.476e+05	1.610e+06	4.568e+06	2.258e+05	0.033	0.0	0.0	0.0	111.523	106.789	1.262e+04	1.642e+03	8.073	3226.045	515.002	1085.593	2.790e+06	0.003	0.033	1.196e-04	0.351	0.504	0.141	0.0	1.196e-04	1.006	574.569	164.827	890.537	111.523	0.0	0.0	0.0	0.0	0.0	0.0	4.875	939.463	4.573	5.960e+03	4247.437	601.049	3.311	21.483	1.848e+05	1.293e+04	2.081e+05	1.776e+05	1.032e+07	3.612e+06	1.288e+07	7.755e+06
std	23057.302	19754.647	3.882	3.152e+07	15.423	21.755	3249.403	3.922e+04	1864.129	163.324	504.893	797.411	3705.123	50.481	1120.325	1733.201	NaN	NaN	2.702e+06	7.623e+06	2.670e+07	7.598e+05	3.161e+07	5.935e+06	1.079e+07	2.749e+07	3.795e+06	2.198e+07	4.586e+06	5.476e+06	1.618e+07	4.019e+06	0.179	0.0	0.0	0.0	375.791	511.766	1.107e+05	1.990e+04	15.768	3813.135	559.064	1269.559	4.116e+06	0.052	0.179	1.094e-02	0.477	0.500	0.348	0.0	1.094e-02	1.431	626.096	504.893	1120.325	375.791	0.0	0.0	0.0	0.0	0.0	0.0	15.423	3249.403	21.755	3.922e+04	8037.781	4319.720	12.270	4.167	7.979e+05	2.103e+05	9.002e+05	7.843e+05	2.185e+07	1.276e+07	2.692e+07	1.983e+07
min	0.000	0.000	0.000	-1.000e+00	1.000	0.000	0.000	0.000e+00	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	-1.200e+07	-2.000e+06	-1.000e+00	0.000e+00	-1.000e+00	-1.200e+01	0.000e+00	0.000e+00	0.000e+00	0.000e+00	-1.200e+01	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000	0.0	0.0	0.0	0.000	0.000	0.000e+00	0.000e+00	0.000	0.000	0.000	0.000	0.000e+00	0.000	0.000	0.000e+00	0.000	0.000	0.000	0.0	0.000e+00	0.000	0.000	0.000	0.000	0.000	0.0	0.0	0.0	0.0	0.0	0.0	1.000	0.000	0.000	0.000e+00	-1.000	-1.000	0.000	0.000	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00
25%	18990.000	80.000	6.000	7.118e+04	2.000	1.000	26.000	0.000e+00	6.000	0.000	6.000	0.000	0.000	0.000	0.000	0.000	1.209e+01	6.186e-01	1.918e+04	1.910e+04	6.118e+04	1.000e+00	1.290e+02	1.100e+02	0.000e+00	1.240e+02	1.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000	0.0	0.0	0.0	64.000	20.000	4.247e-01	3.527e-02	0.000	6.000	6.000	0.000	0.000e+00	0.000	0.000	0.000e+00	0.000	0.000	0.000	0.0	0.000e+00	0.000	7.500	6.000	0.000	64.000	0.0	0.0	0.0	0.0	0.0	0.0	2.000	26.000	1.000	0.000e+00	229.000	-1.000	1.000	20.000	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00
50%	49799.000	80.000	6.000	1.452e+06	3.000	4.000	30.000	1.640e+02	20.000	0.000	8.667	5.302	99.000	0.000	92.000	2.449	1.137e+03	5.179e+00	2.245e+05	5.642e+05	1.423e+06	3.000e+00	2.371e+04	1.033e+04	1.273e+04	2.303e+04	4.000e+00	4.110e+04	1.001e+04	1.470e+04	3.336e+04	3.000e+00	0.000	0.0	0.0	0.0	72.000	92.000	2.316e+00	1.483e+00	0.000	513.000	110.333	154.031	2.373e+04	0.000	0.000	0.000e+00	0.000	1.000	0.000	0.0	0.000e+00	1.000	141.000	8.667	92.000	72.000	0.0	0.0	0.0	0.0	0.0	0.0	3.000	30.000	4.000	1.640e+02	256.000	126.000	2.000	20.000	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00	0.000e+00
75%	58296.000	80.000	6.000	8.805e+06	5.000	5.000	63.000	1.160e+04	34.000	6.000	32.000	10.263	5840.000	6.000	1934.500	2436.833	2.160e+04	7.047e+01	2.013e+06	4.033e+06	8.359e+06	3.900e+01	7.870e+06	2.018e+06	3.680e+06	7.548e+06	1.220e+02	1.112e+06	2.656e+05	5.178e+05	1.094e+06	4.700e+01	0.000	0.0	0.0	0.0	100.000	120.000	3.287e+01	2.102e+01	6.000	5840.000	1162.700	2131.871	4.545e+06	0.000	0.000	0.000e+00	1.000	1.000	0.000	0.0	0.000e+00	1.000	1291.889	32.000	1934.500	100.000	0.0	0.0	0.0	0.0	0.0	0.0	5.000	63.000	5.000	1.160e+04	8192.000	229.000	4.000	20.000	1.878e+03	0.000e+00	1.878e+03	1.862e+03	8.240e+06	0.000e+00	8.254e+06	7.423e+06
max	65534.000	65532.000	17.000	1.200e+08	1932.000	2942.000	183012.000	5.172e+06	11680.000	1472.000	3867.000	6692.645	11680.000	1460.000	5800.500	8194.660	inf	inf	1.070e+08	6.920e+07	1.200e+08	1.070e+08	1.200e+08	1.200e+08	7.670e+07	1.200e+08	1.200e+08	1.200e+08	1.200e+08	7.670e+07	1.200e+08	1.200e+08	1.000	0.0	0.0	0.0	39396.000	58852.000	3.000e+06	2.000e+06	337.000	11680.000	1936.833	4731.522	2.240e+07	1.000	1.000	1.000e+00	1.000	1.000	1.000	0.0	1.000e+00	7.000	2528.000	3867.000	5800.500	39396.000	0.0	0.0	0.0	0.0	0.0	0.0	1932.000	183012.000	2942.000	5.172e+06	65535.000	65535.000	1931.000	52.000	1.000e+08	3.950e+07	1.000e+08	1.000e+08	1.200e+08	6.530e+07	1.200e+08	1.200e+08

	Feature 1	Feature 2	Correlation
121	Fwd Header Length	Fwd Header Length.1	1.000
119	Fwd PSH Flags	SYN Flag Count	1.000
34	Total Length of Fwd Packets	Subflow Fwd Bytes	1.000
146	RST Flag Count	ECE Flag Count	1.000
40	Total Length of Bwd Packets	Subflow Bwd Bytes	1.000
18	Total Fwd Packets	Subflow Fwd Packets	1.000
47	Fwd Packet Length Mean	Avg Fwd Segment Size	1.000
27	Total Backward Packets	Subflow Bwd Packets	1.000
66	Bwd Packet Length Mean	Avg Bwd Segment Size	1.000
139	Packet Length Mean	Average Packet Size	0.999
95	Flow IAT Max	Idle Max	0.997
7	Flow Duration	Fwd IAT Total	0.997
93	Flow IAT Max	Fwd IAT Max	0.995
111	Fwd IAT Max	Idle Max	0.993
52	Bwd Packet Length Max	Bwd Packet Length Std	0.993
43	Fwd Packet Length Max	Fwd Packet Length Std	0.992
132	Max Packet Length	Packet Length Std	0.984
73	Flow Packets/s	Fwd Packets/s	0.984
162	Active Mean	Active Min	0.983
107	Fwd IAT Std	Fwd IAT Max	0.980
82	Flow IAT Std	Flow IAT Max	0.978
88	Flow IAT Std	Idle Max	0.976
24	Total Backward Packets	Bwd Header Length	0.976
128	Bwd Header Length	Subflow Bwd Packets	0.976
92	Flow IAT Max	Fwd IAT Std	0.972
39	Total Length of Bwd Packets	Subflow Bwd Packets	0.970
28	Total Backward Packets	Subflow Bwd Bytes	0.970
158	Subflow Bwd Packets	Subflow Bwd Bytes	0.970
22	Total Backward Packets	Total Length of Bwd Packets	0.970
109	Fwd IAT Std	Idle Max	0.970
86	Flow IAT Std	Fwd IAT Max	0.969
122	Fwd Header Length	Subflow Fwd Packets	0.968
151	Fwd Header Length.1	Subflow Fwd Packets	0.968
15	Total Fwd Packets	Fwd Header Length	0.968
17	Total Fwd Packets	Fwd Header Length.1	0.968
51	Bwd Packet Length Max	Bwd Packet Length Mean	0.961
58	Bwd Packet Length Max	Avg Bwd Segment Size	0.961
118	Bwd IAT Std	Bwd IAT Max	0.959
26	Total Backward Packets	Subflow Fwd Packets	0.957
19	Total Fwd Packets	Subflow Bwd Packets	0.957
155	Subflow Fwd Packets	Subflow Bwd Packets	0.957
13	Total Fwd Packets	Total Backward Packets	0.957
60	Bwd Packet Length Mean	Bwd Packet Length Std	0.956
72	Bwd Packet Length Std	Avg Bwd Segment Size	0.956
164	Idle Mean	Idle Max	0.953
120	Fwd Header Length	Bwd Header Length	0.953
126	Bwd Header Length	Fwd Header Length.1	0.953
136	Packet Length Mean	Packet Length Std	0.951
142	Packet Length Std	Average Packet Size	0.950
141	Packet Length Std	Packet Length Variance	0.950
94	Flow IAT Max	Idle Mean	0.950
129	Bwd Header Length	Subflow Bwd Bytes	0.945
36	Total Length of Bwd Packets	Bwd Header Length	0.945
42	Fwd Packet Length Max	Fwd Packet Length Mean	0.941
44	Fwd Packet Length Max	Avg Fwd Segment Size	0.941
110	Fwd IAT Max	Idle Mean	0.940
152	Fwd Header Length.1	Subflow Bwd Packets	0.940
123	Fwd Header Length	Subflow Bwd Packets	0.940
25	Total Backward Packets	Fwd Header Length.1	0.940
23	Total Backward Packets	Fwd Header Length	0.940
14	Total Fwd Packets	Total Length of Bwd Packets	0.938
38	Total Length of Bwd Packets	Subflow Fwd Packets	0.938
156	Subflow Fwd Packets	Subflow Bwd Bytes	0.938
20	Total Fwd Packets	Subflow Bwd Bytes	0.938
87	Flow IAT Std	Idle Mean	0.938
85	Flow IAT Std	Fwd IAT Std	0.936
161	Active Mean	Active Max	0.934
133	Max Packet Length	Packet Length Variance	0.930
127	Bwd Header Length	Subflow Fwd Packets	0.929
16	Total Fwd Packets	Bwd Header Length	0.929
124	Fwd Header Length	Subflow Bwd Bytes	0.923
37	Total Length of Bwd Packets	Fwd Header Length.1	0.923
35	Total Length of Bwd Packets	Fwd Header Length	0.923
153	Fwd Header Length.1	Subflow Bwd Bytes	0.923
99	Fwd IAT Total	Fwd IAT Max	0.922
77	Flow IAT Mean	Fwd IAT Mean	0.921
6	Flow Duration	Flow IAT Max	0.920
49	Fwd Packet Length Std	Avg Fwd Segment Size	0.920
46	Fwd Packet Length Mean	Fwd Packet Length Std	0.920
12	Flow Duration	Idle Max	0.919
10	Flow Duration	Fwd IAT Max	0.918
74	Flow IAT Mean	Flow IAT Std	0.918
90	Flow IAT Max	Fwd IAT Total	0.917
101	Fwd IAT Total	Idle Max	0.917
108	Fwd IAT Std	Idle Mean	0.913
165	Idle Mean	Idle Min	0.911
114	Fwd IAT Min	Bwd IAT Min	0.906
131	Max Packet Length	Packet Length Mean	0.904
134	Max Packet Length	Average Packet Size	0.900
98	Fwd IAT Total	Fwd IAT Std	0.900
5	Flow Duration	Flow IAT Std	0.899
9	Flow Duration	Fwd IAT Std	0.896
3	Protocol	Min Packet Length	0.895
83	Flow IAT Std	Fwd IAT Total	0.894
69	Bwd Packet Length Std	Packet Length Std	0.889
55	Bwd Packet Length Max	Packet Length Std	0.886
21	Total Fwd Packets	act_data_pkt_fwd	0.882
157	Subflow Fwd Packets	act_data_pkt_fwd	0.882
53	Bwd Packet Length Max	Max Packet Length	0.878
84	Flow IAT Std	Fwd IAT Mean	0.876
70	Bwd Packet Length Std	Packet Length Variance	0.875
67	Bwd Packet Length Std	Max Packet Length	0.874
140	Packet Length Mean	Avg Bwd Segment Size	0.872
62	Bwd Packet Length Mean	Packet Length Mean	0.872
11	Flow Duration	Idle Mean	0.872
117	Bwd IAT Mean	Bwd IAT Min	0.870
149	Average Packet Size	Avg Bwd Segment Size	0.870
65	Bwd Packet Length Mean	Average Packet Size	0.870
163	Active Max	Active Min	0.868
100	Fwd IAT Total	Idle Mean	0.868
29	Total Backward Packets	act_data_pkt_fwd	0.865
159	Subflow Bwd Packets	act_data_pkt_fwd	0.865
63	Bwd Packet Length Mean	Packet Length Std	0.863
143	Packet Length Std	Avg Bwd Segment Size	0.863
33	Total Length of Fwd Packets	Avg Fwd Segment Size	0.858
31	Total Length of Fwd Packets	Fwd Packet Length Mean	0.858
48	Fwd Packet Length Mean	Subflow Fwd Bytes	0.858
150	Avg Fwd Segment Size	Subflow Fwd Bytes	0.858
56	Bwd Packet Length Max	Packet Length Variance	0.855
45	Fwd Packet Length Max	Subflow Fwd Bytes	0.852
30	Total Length of Fwd Packets	Fwd Packet Length Max	0.852
75	Flow IAT Mean	Flow IAT Max	0.851
144	Packet Length Variance	Average Packet Size	0.851
137	Packet Length Mean	Packet Length Variance	0.845
116	Bwd IAT Total	Bwd IAT Max	0.845
81	Flow IAT Mean	Idle Max	0.842
79	Flow IAT Mean	Fwd IAT Max	0.841
32	Total Length of Fwd Packets	Fwd Packet Length Std	0.839
50	Fwd Packet Length Std	Subflow Fwd Bytes	0.839
41	Total Length of Bwd Packets	act_data_pkt_fwd	0.839
160	Subflow Bwd Bytes	act_data_pkt_fwd	0.839
61	Bwd Packet Length Mean	Max Packet Length	0.833
135	Max Packet Length	Avg Bwd Segment Size	0.833
54	Bwd Packet Length Max	Packet Length Mean	0.833
57	Bwd Packet Length Max	Average Packet Size	0.833
71	Bwd Packet Length Std	Average Packet Size	0.824
68	Bwd Packet Length Std	Packet Length Mean	0.823
115	Bwd IAT Total	Bwd IAT Std	0.821
103	Fwd IAT Mean	Fwd IAT Max	0.819
80	Flow IAT Mean	Idle Mean	0.814
91	Flow IAT Max	Fwd IAT Mean	0.812
106	Fwd IAT Mean	Idle Max	0.807
4	Flow Duration	Flow IAT Mean	0.798
76	Flow IAT Mean	Fwd IAT Total	0.793
113	Fwd IAT Min	Bwd IAT Mean	0.788
78	Flow IAT Mean	Fwd IAT Std	0.783
64	Bwd Packet Length Mean	Packet Length Variance	0.781
145	Packet Length Variance	Avg Bwd Segment Size	0.781
105	Fwd IAT Mean	Idle Mean	0.779
125	Fwd Header Length	act_data_pkt_fwd	0.776
154	Fwd Header Length.1	act_data_pkt_fwd	0.776
97	Fwd IAT Total	Fwd IAT Mean	0.770
8	Flow Duration	Fwd IAT Mean	0.766
148	PSH Flag Count	Average Packet Size	0.760
102	Fwd IAT Mean	Fwd IAT Std	0.759
138	Packet Length Mean	PSH Flag Count	0.755
130	Bwd Header Length	act_data_pkt_fwd	0.753
1	Destination Port	URG Flag Count	0.751
89	Flow IAT Std	Idle Min	0.743
166	Idle Max	Idle Min	0.742
96	Flow IAT Max	Idle Min	0.739
104	Fwd IAT Mean	Bwd IAT Mean	0.733
147	PSH Flag Count	ACK Flag Count	-0.728
0	Source Port	Destination Port	-0.727
59	Bwd Packet Length Min	Min Packet Length	0.727
2	Protocol	Bwd Packet Length Min	0.726
112	Fwd IAT Max	Idle Min	0.724

	eps	min_samples	n_clusters	noise_pct	silhouette
15	1.0	50	13	18.97	0.698
7	0.5	50	8	31.40	0.689
11	0.7	50	12	23.69	0.612
14	1.0	20	20	12.00	0.592
3	0.3	50	14	45.16	0.589

	eps	min_samples	n_clusters	n_noise	noise_pct	silhouette	time_sec
0	0.3	5	98	1940	19.40	0.259	0.450
1	0.3	10	53	2442	24.42	0.337	0.364
2	0.3	20	32	3282	32.82	0.435	0.312
3	0.3	50	14	4516	45.16	0.589	0.240
4	0.5	5	78	1265	12.65	0.500	0.492
5	0.5	10	41	1731	17.31	0.530	0.463
6	0.5	20	19	2269	22.69	0.572	0.410
7	0.5	50	8	3140	31.40	0.689	0.350
8	0.7	5	69	859	8.59	0.528	0.549
9	0.7	10	41	1213	12.13	0.543	0.517
10	0.7	20	25	1705	17.05	0.550	0.473
11	0.7	50	12	2369	23.69	0.612	0.419
12	1.0	5	63	563	5.63	0.549	0.611
13	1.0	10	39	825	8.25	0.572	0.583
14	1.0	20	20	1200	12.00	0.592	0.551
15	1.0	50	13	1897	18.97	0.698	0.495
16	1.5	5	45	363	3.63	0.465	0.686
17	1.5	10	30	502	5.02	0.477	0.661
18	1.5	20	19	798	7.98	0.509	0.623
19	1.5	50	12	1231	12.31	0.580	0.591

True Label	BENIGN	DDoS
Cluster
0	264	127
1	1028	476
2	20784	27597
3	111	47
4	13	10
...	...	...
289	35	0
290	9	0
291	17	0
292	24	0
293	8	3

True Label	BENIGN	DDoS
Cluster
0	67.5	32.5
1	68.4	31.6
2	43.0	57.0
3	70.3	29.7
4	56.5	43.5
...	...	...
289	100.0	0.0
290	100.0	0.0
291	100.0	0.0
292	100.0	0.0
293	72.7	27.3

	DBSCAN	Hierarchical
silhouette	0.355	0.408
davies_bouldin	0.519	1.151
calinski_harabasz	19274.958	1684.970
adjusted_rand_index	0.207	0.110
normalized_mutual_info	0.331	0.215

	Port	Service	Benign_Count	Benign_Pct	Total	Difference
22	80	HTTP	33525	34.308	33525	-34.308
16	443	HTTPS	6419	6.569	6419	-6.569
39	123	NTP	362	0.370	362	-0.370
2	137	NetBIOS_Name	274	0.280	274	-0.280
7	22	SSH	156	0.160	156	-0.160
6	21	FTP_Control	86	0.088	86	-0.088
0	0	Unknown	54	0.055	54	-0.055
3	138	NetBIOS_Datagram	53	0.054	53	-0.054
4	139	NetBIOS_Session	31	0.032	31	-0.032
1	389	LDAP	26	0.027	26	-0.027
28	39524	Unknown	20	0.020	20	-0.020
18	48318	Unknown	20	0.020	20	-0.020
29	5353	Unknown	16	0.016	16	-0.016
17	445	SMB	14	0.014	14	-0.014
24	88	Unknown	14	0.014	14	-0.014
21	9292	Unknown	14	0.014	14	-0.014
19	3268	Unknown	14	0.014	14	-0.014
27	56540	Unknown	14	0.014	14	-0.014
38	60406	Unknown	14	0.014	14	-0.014
23	60883	Unknown	13	0.013	13	-0.013

	Overall %	Noise %	Difference
Label
BENIGN	43.29	97.82	54.53
DDoS	56.71	2.18	-54.53

	Feature	Benign_Mean	Attack_Mean	Abs_Difference	Cohens_d	Separation_Ratio
14	Bwd Packet Length Mean	116.896	1.481e+03	1.364e+03	1.617	0.921
56	Avg Bwd Segment Size	116.896	1.481e+03	1.364e+03	1.617	0.921
12	Bwd Packet Length Max	287.173	4.604e+03	4.317e+03	1.514	0.938
15	Bwd Packet Length Std	87.161	2.103e+03	2.015e+03	1.512	0.959
1	Destination Port	20406.992	8.123e+01	2.033e+04	1.113	0.996
42	Packet Length Mean	224.298	7.369e+02	5.126e+02	1.061	0.696
54	Average Packet Size	249.591	8.226e+02	5.730e+02	1.060	0.697
43	Packet Length Std	440.752	1.578e+03	1.137e+03	1.029	0.721
2	Protocol	9.697	6.000e+00	3.697e+00	1.005	0.381
50	URG Flag Count	0.325	2.343e-05	3.251e-01	0.981	1.000
41	Max Packet Length	1417.359	4.607e+03	3.189e+03	0.941	0.692
44	Packet Length Variance	867317.091	4.257e+06	3.390e+06	0.937	0.796
40	Min Packet Length	15.786	2.185e+00	1.360e+01	0.890	0.862
71	min_seg_size_forward	23.425	2.000e+01	3.425e+00	0.838	0.146
0	Source Port	27998.474	4.609e+04	1.809e+04	0.822	0.392

	Model	Sample_Time	Projected_Single	Projected_10fold_CV
0	Logistic Regression	0.380	11.490	114.897
1	Random Forest	0.275	8.315	83.154
2	XGBoost	0.167	5.053	50.534
3	SVM (RBF)	1.336	3235.845	32358.450

	File	Samples	Features	Benign	Attack	Imbalance
0	Friday-WorkingHours-Afternoon-PortScan.pcap_IS...	286467	85	127537	158930	0.80
1	Friday-WorkingHours-Morning.pcap_ISCX.csv	191033	85	189067	1966	96.17
2	Monday-WorkingHours.pcap_ISCX.csv	529918	85	529918	0	inf
3	Thursday-WorkingHours-Afternoon-Infilteration....	288602	85	288566	36	8015.72
4	Tuesday-WorkingHours.pcap_ISCX.csv	445909	85	432074	13835	31.23
5	Wednesday-workingHours.pcap_ISCX.csv	692703	85	440031	252672	1.74

	Port	Service	Benign_Count	Benign_Pct	Attack_Count	Attack_Pct	Total	Difference
10	80	HTTP	8927	9.135	128024	99.998	136951	90.862
20	53	DNS	31950	32.696	0	0.000	31950	-32.696
21	443	HTTPS	13485	13.800	0	0.000	13485	-13.800
9	8080	HTTP_Proxy	510	0.522	0	0.000	510	-0.522
18	123	NTP	362	0.370	0	0.000	362	-0.370
13	22	SSH	342	0.350	0	0.000	342	-0.350
6	137	NetBIOS_Name	274	0.280	0	0.000	274	-0.280
3	389	LDAP	261	0.267	0	0.000	261	-0.267
14	88	Unknown	173	0.177	0	0.000	173	-0.177
12	21	FTP_Control	167	0.171	0	0.000	167	-0.171
11	465	Unknown	147	0.150	0	0.000	147	-0.150
8	139	NetBIOS_Session	100	0.102	0	0.000	100	-0.102
2	3268	Unknown	91	0.093	0	0.000	91	-0.093
0	0	Unknown	54	0.055	0	0.000	54	-0.055
7	138	NetBIOS_Datagram	53	0.054	0	0.000	53	-0.054
22	445	SMB	48	0.049	0	0.000	48	-0.049
1	49666	Unknown	23	0.024	0	0.000	23	-0.024
4	135	Unknown	23	0.024	0	0.000	23	-0.024
16	5353	Unknown	16	0.016	0	0.000	16	-0.016
5	49671	Unknown	12	0.012	0	0.000	12	-0.012

	Model	Accuracy	Precision	Recall	F1	Roc_auc
0	Logistic Regression	0.9993 ± 0.0001	0.9997 ± 0.0002	0.9992 ± 0.0002	0.9994 ± 0.0001	0.9999 ± 0.0001
1	Random Forest	0.9999 ± 0.0000	1.0000 ± 0.0000	0.9999 ± 0.0001	0.9999 ± 0.0000	1.0000 ± 0.0000
2	XGBoost	1.0000 ± 0.0000	1.0000 ± 0.0000	0.9999 ± 0.0001	1.0000 ± 0.0000	1.0000 ± 0.0000

	File	DBSCAN_Clusters	DBSCAN_Noise%
0	Friday-WorkingHours-Afternoon-PortScan.pcap_IS...	434	7.6%
1	Friday-WorkingHours-Morning.pcap_ISCX.csv	452	11.4%
2	Monday-WorkingHours.pcap_ISCX.csv	679	8.0%
3	Thursday-WorkingHours-Afternoon-Infilteration....	582	9.2%
4	Tuesday-WorkingHours.pcap_ISCX.csv	537	8.5%
5	Wednesday-workingHours.pcap_ISCX.csv	759	3.8%

	File	XGB_Accuracy	XGB_Precision	XGB_Recall	XGB_F1	XGB_ROC_AUC
0	Friday-WorkingHours-Afternoon-PortScan.pcap_IS...	1.0000	1.0000	0.9999	1.0000	1.0000
1	Friday-WorkingHours-Morning.pcap_ISCX.csv	0.9994	0.9524	0.9944	0.9729	0.9999
2	Monday-WorkingHours.pcap_ISCX.csv	1.0000	0.0000	0.0000	0.0000	nan
3	Thursday-WorkingHours-Afternoon-Infilteration....	1.0000	0.9083	0.7833	0.8171	0.9413
4	Tuesday-WorkingHours.pcap_ISCX.csv	0.9999	0.9987	0.9993	0.9990	1.0000
5	Wednesday-workingHours.pcap_ISCX.csv	0.9994	0.9987	0.9998	0.9992	1.0000

Network Intrusion Detection: A Complete Machine Learning Pipeline¶

Executive Summary¶

Key Results¶

Table of Contents¶

1. Business Problem and Context¶

Problem Statement¶

Dataset: CIC-IDS-2017¶

Dataset Advantages¶

Success Criteria¶

Real-World Impact¶

2. Technical Stack¶

Programming Languages & Core Libraries¶

Machine Learning Techniques Implemented¶

Computational Environment¶

Methodology Note: Model Selection¶

Computational Feasibility Analysis¶

Decision Rationale¶

Questions 1 and 2¶

Pipeline Structure¶

Setup: Import Required Libraries¶

Stage 1: Data Loading¶

Stage 2: Initial Inspection¶

Stage 3: Data Quality Check¶

Stage 4: Statistical Summary¶

Stage 5: Missing Data Visualization¶

Stage 6: Distribution Analysis¶

Box Plots for Outlier Detection¶

Stage 7: Correlation Analysis¶

Optional: View All Correlations or Customize Threshold¶

Stage 8: Comprehensive Report¶

Optional: Export Summary Statistics¶

Question 3: UNSUPERVISED LEARNING - CLUSTERING ANALYSIS¶

Methodology Selection¶

Why Unsupervised Learning?¶

Method 1: DBSCAN (Primary)¶

Method 2: Hierarchical Clustering (Comparative)¶

Step 1: Data Preprocessing for Clustering¶

Step 2: Dimensionality Reduction (Optional but Recommended)¶

Step 3: DBSCAN Clustering¶

Step 4: Hierarchical Clustering¶

⚠️ IMPORTANT: Scalability Limitation¶

Recommendation for analysis¶

Linkage Methods¶

Step 5: Cluster Evaluation and Comparison¶

Internal Metrics (don't use labels):¶

External Metrics (use labels for validation):¶

Step 6: Compare Methods¶

Step 7: Analyze Noise Points (DBSCAN)¶

Step 8: Summary and Recommendations¶

Question 4: SUPERVISED LEARNING PREPARATION¶

Binary Classification Setup¶

Task: Binary Classification¶

Class Balance Analysis¶

Implications for Machine Learning¶

Save Prepared Dataset (Optional)¶

Question 5¶

Feature Engineering: Port Number Analysis¶

Why Port Numbers Matter in Network Security¶

Analyze Port Distribution in Dataset¶

Feature Engineering: One-Hot Encoding Ports¶

Port Analysis by Class (BENIGN vs ATTACK)¶

Feature Importance: Port Features vs Class¶

Summary: Port Feature Engineering Results¶

Question 6¶

Feature Distribution Analysis by Class¶

Purpose: Compare BENIGN vs ATTACK Traffic¶

1. Bytes Features: Compare Data Volume¶

2. Packets Features: Compare Traffic Volume¶

3. Duration Features: Compare Flow Length¶

4. Comprehensive Box Plot Comparison¶

5. Summary: Most Discriminative Features¶

Question 7: SUPERVISED LEARNING - BINARY CLASSIFICATION¶

Model Training and Evaluation¶

Step 1: Feature Preparation¶

Understanding Model Evaluation Metrics¶

Classification Metrics Explained¶

Confusion Matrix Components¶

Key Metrics¶

Why These Metrics Matter for Intrusion Detection¶

Step 2: Cross-Validation Setup¶